English Wikipedia @ Freddythechick:Bots/Requests for approval/ThePhantomBot

Add links
From English Wikipedia @ Freddythechick
The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was Request Expired.

ThePhantomBot

Operator: PhantomTech (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 02:11, Thursday, March 19, 2015 (UTC)

Automatic, Supervised, or Manual: Automatic

Programming language(s): Python

Source code available: No, not now at least, though I'll share some of the regex if asked during the approval process

Function overview: Monitors recent changes for possible vandalism and edits from long term abuse users, logs findings and (sometimes) gives information to AN/I for review by users.

Links to relevant discussions (where appropriate): Not sure if this would require consensus from AN/I since it would be posting there or not since the posting task is simple and likely to be uncontroversial.

Edit period(s): daily (while I have a computer on) with plans to make it continuous

Estimated number of pages affected: 1 (AN/I) not counting pages in its own user space

Exclusion compliant (Yes/No): no

Already has a bot flag (Yes/No): no

Function details: This bot is meant to allow a decrease in the amount of edit filters and to identify abuse that can't be reverted by bots like ClueBot due to lack of certainty. Every 60 seconds (that time might be lowered to 20-40 seconds to spread load) a list of changes since the time of the last check is filled. On a separate thread, the bot goes through the list, and decides if the actions match a set filter, these filters are usually similar in what they check to the edit filters however are not limited to the same restraints. If a filter is matched the associated actions are taken, usually logging to the user space and sometimes a noticeboard report. Essentially, this bot acts as a post-edit filter, currently targeting long term abuse but technically able to act on any identifiable action. Since it happens after edits, as opposed to "during" edits, it doesn't slow down editing for users so problematic edits don't have to be frequent, like they do to be edit filter worthy, for it to be worth it for this bot to check for them. In its current state I have two LTA matches setup, one stolen from a log only edit filter and another stolen from edit filter requests, and a general abuse match, also stolen from edit filter requests. If the bot is accepted, I plan on going through all the active long term abuse cases and adding whichever ones I can along with some edit filter requests that aren't accepted due to infrequency.

Discussion

Vandalism/abuse monitoring is a difficult area; I suggest that you write your bot and have it edit a page in its or your userspace (no approval necessary unless edit rates are high) as if it were ANI, and monitor what it reports. You can in turn pass the valid observations it makes along to ANI, and if the quality of the reporting is high enough you may find other people watching the page to see what it finds. I expect you'll get a high false-positive rate which you'll need to analyse to improve the performance of your algorithms, and eventually you'll get to a point where regexs just don't cut it for detecting the long-term, low-frequency abuse you're targetting - and you'll have to look at more sophisticated processes. This is the technological evolution that Cluebot went through, but it catches more egregious and obvious vandalism.

Do you think operating in your or the bot's own userspace would be an acceptable stepping stone? Josh Parris 22:18, 20 March 2015 (UTC)

I realize that there is lots of long term abuse that can't be solved by regex alone, this bot will never be able to handle every LTA case but I do plan on implementing more advanced checks in the future. I have no problem running my bot for a bit with it doing nothing but logging to User:ThePhantomBot/log. PhantomTech (talk) 22:36, 20 March 2015 (UTC)
I would want to see a community consensus that bot generated ANI reports are wanted, please discuss and link that discussion here. — xaosflux Talk 05:43, 26 March 2015 (UTC)
@Xaosflux: As I've been working on my bot I've been adding more functionality and thinking about the best ways to have the bot's reports dealt with. Here's my current plan for how it will report things:
  • Bad page recreation - Log to user space
  • High probability sockpuppets - Report to SPI
  • Lower probability sockpuppets - Log to user space
  • LTA detection - Report to AIV or report to AN/I where certainty is reasonably low (not too low, don't want to waste people's time)
  • Newly added LTA filters, including ones being tested - Log to user space
  • IPs using administrative templates - Report to AN/I
  • Sleeper account detection - Not implemented yet so I don't know how often it will go off, if its often log to user space otherwise report to AN/I
I assume you still want to see a discussion for the AN/I reports but do you want to see any for the other places? I'm guessing you'll want SPI mentioned in the discussion too since I don't think any bots currently report to there. Also, do you have any suggestions on where to report these things or how to report them? Admittedly AN/I does feel like a weird place for bot reports but the goal is to get the attention of editors who may not be aware of the bot's existence. PhantomTech (talk) 07:03, 26 March 2015 (UTC)
Start reading AIV archvies such as Wikipedia_talk:Administrator_intervention_against_vandalism/Archive_3#Suggested_merge_of_Wikipedia:Administrator_intervention_against_vandalism.2FTB2 for some suggestions. WP:AIV/TB2 is probably the oldest 'bot reported' noticeboard right now. — xaosflux Talk 10:23, 26 March 2015 (UTC)
@Xaosflux: Are you suggesting that if my bot were to report to ANI it should do so via a transuded page? I like that idea, using transclusion to put the bot's reports somewhere they'll be seen keeps the bot's updates off the watchlists of people who don't care. PhantomTech (talk) 15:32, 26 March 2015 (UTC)
I'm sugesting that prior community discussion on ANI bot reports came to that conclusion - and that after reading up on it you start new discussions to find out where people would make best use of your reports. For ANI it could be the existing TB2 subpage, but they might want it on its OWN subpage; for the other forums people might want subpages, might want main, or might not want bot reports at all. I am not trying to dictate the solution, just that whatever it is should enjoy community consensus before integrating to existing forums. — xaosflux Talk 16:59, 26 March 2015 (UTC)

I posted a discussion at Wikipedia:Village_pump_(idea_lab)#ThePhantomBot_reporting_to_noticeboards to help get an idea of what kind of reporting users would like. Depending on how that goes I'll post something to village pump proposals with notifications on the relevant noticeboard's talk pages. PhantomTech (talk) 05:57, 27 March 2015 (UTC)


There's a lot of room for fuzzy logic on some of the things you've mentioned. To name a few:
  • High probability sockpuppets / Lower probability sockpuppets — exactly how are these detected?
  • Bad page recreation — what's detected here?
  • LTA detection — how does this magic work? Do you have any code already written, and if so, which forms of LTA did you have in mind? Any examples?
A huge bonus of the abuse filter is that people can actually see what the filters are detecting and make tweaks if there are a crapton of false positives, for example.
Also, any updates otherwise on the stuff raised above? It looks like this has been sitting here for a few months. :P
--slakr\ talk / 05:48, 4 June 2015 (UTC)
@Slakr: Sorry about how long this has been sitting here, I've gotten busy since starting it which has slowed it down a bit, hopefully I'll be able to get more time to work on it more often soon. To your concerns about "fuzziness", you're right, there aren't any clear cut ways to tell what is considered, for example, "high probability" vs "low probability" which means that, to some extent, my judgment will have to be trusted in deciding where some things go. I don't expect too many people to just trust me without having something to look at so I think the best way to gain support for things outside of the bot's userspace is to be able to show some examples of real detections and where they would be reported. Currently, almost all detections are sent to the debug log, but I'm working on setting up different logging locations and don't plan on seeking consensus until after those have been reported to for a bit.
Sockpuppets: I'm working on different ways to detect "general" (non-LTA case) sockpuppets, what the bot can detect right now is based on removing speedy deletion tags and I don't think it causes enough disruption to be worth an SPI. I'm not sure what other ways I'll come up with for general sockpuppet detection so, if the bot ends up being allowed to report to SPI, the early reports will all be LTA cases with new reports only happening after several successful manual reports.
LTA: LTA cases are detected using anything from the same basic checks an abuse filter can do to ones that involve looking through all the user's contributions for patterns or past behavior. The ones that can be detected are any that I'm aware of and can think of a way to detect, which would include any that currently have abuse filters setup for them. I have detections setup for a few cases (see User:ThePhantomBot/Edit Codes#LTA: Long Term Abuse for a list) but a lot of them haven't been updated during my inactivity.
Bad pages: Bad page detections are pages that probably have to be deleted, (or moved) currently bad recreates are the only things detected (and reported here) but I plan on setting up something to detect autobiographies better than the abuse filters do and have something "ready" to detect self-published drafts.
Help from other users: I agree that a bonus to the abuse filters is that more than one user can edit them, since that isn't as easy with a bot I think it's something that will limit its usefulness a bit so I hope someone will be able to think of some way to allow others to help more directly without having to make all the detection algorithms public. Until then, I don't mind sharing detection algorithms with anyone who is currently trusted by the community and has a reason to see them (like abuse filter managers) via email or something. I'd also like to keep abuse filters setup and updated (but disabled) that would exist if my bot didn't, so they can easily be enabled in case something happens to my bot. Keeping the disabled abuse filters updated is something I wouldn't mind doing personally (if I can get the perm) or by working with abuse filter managers. To mitigate the issue of something causing false positive spam, I plan to have a fully protected page to allow specific detections to be turned off by administrators without having to turn them all off by blocking or something and testing detections in the debug log before having them file any reports.
Other Updates: Sleeper account detection is something I have a few ideas for how to do but haven't worked on in a while, it probably won't be done till a while after everything else has been. I've been working on setting up what's needed for the ANI reports ("priority" reports) and I think the proposal will end up being to transclude a navbox (this one, once it's done) so that it is split from the actual noticeboard page. ANI is still the best place I can think of to have the "priority" reports go since the goal is to get experienced editors to see them soon after being reported and decide what (if anything) needs to be done, and I think using a navbox and transclusion will allow for minimal disruption, but I'm open to other ideas and realize it may not get enough support to happen.
Hopefully that answered all your questions without being too long, if not, let me know. PHANTOMTECH (talk) 07:43, 4 June 2015 (UTC)
A user has requested the attention of the operator. Once the operator has seen this message and replied, please deactivate this tag. (user notified) There seem to be a lot of false positives or things I'm not sure about:
  • [1] — not sure why this is problematic.
  • 2015-07-08 07:00:19 Detected UA-6, pageid: 47183859 [2] — At best I can tell, this is solely because the user moved a draft-space article into mainspace. How/where would that be reported, and why?
  • [3] - appears to be a new page, but the prior delete was for A7?
These are only some of the ones I checked. There are some good detections in my sample, but false positives—rather, the proportion of them and the assumptions made—can be more problematic with the community. Granted, all these were well over a month ago, and it seems like you've been a little inactive (not to mention, nobody else has chimed in here in the meantime :P). Have there been any updates in the meantime? --slakr\ talk / 04:59, 27 August 2015 (UTC)

Request Expired. for now; can revisit when the operator becomes active again. --slakr\ talk / 23:19, 2 September 2015 (UTC)

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.
Retrieved from "https://enwiki.freddythechick.net/index.php?title=English_Wikipedia_@_Freddythechick:Bots/Requests_for_approval/ThePhantomBot&oldid=5039536"