b3yond/ticketfrei
1
0
Fork 0
Code Issues 28 Pull requests Releases Wiki Activity

merged ticketfrei 1.0 into multi-deployment #3

Browse source
This commit is contained in:
b3yond 2018-01-19 16:33:46 +01:00
parent 9ef0b27970 c0328be3a4
commit 63cf134ffa
8 changed files with 297 additions and 160 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

86
README.md
View file

@ -1,21 +1,33 @@
# Ticketfrei micro messaging bot
Version: 1.0
<!-- This mastodon/twitter bot has one purpose - breaking the law. -->
The functionality is simple: it retweets every tweet where it is mentioned.
The functionality is simple: it retweets every tweet where it is
mentioned.
This leads to a community which evolves around it; if you see ticket controllers, you tweet their location and mention the bot. The bot then retweets your tweet and others can read the info and think twice if they want to buy a ticket. If enough people, a critical mass, participate for the bot to become reliable, you have positive self-reinforcing dynamics.
This leads to a community which evolves around it; if you see ticket
controllers, you tweet their location and mention the bot. The bot
then retweets your tweet and others can read the info and think twice
if they want to buy a ticket. If enough people, a critical mass,
participate for the bot to become reliable, you have positive
self-reinforcing dynamics.
There is one security hole: people could start mentioning the bot with useless information, turning it into a spammer. That's why it has to be maintained; if someone spams the bot, mute them and undo the retweet. So it won't retweet their future tweets and the useless retweet is deleted if someone tries to check if something was retweeted in the last hour or something.
To this date, we have never heard of this happening though.
In the promotion folder, you will find some promotion material you
can use to build up such a community in your city. It is in german
though =/
Website: https://wiki.links-it.de/IT/Ticketfrei
Website: https://wiki.links-tech.org/IT/Ticketfrei
## Install
Setting up a ticketfrei bot for your city is quite easy. Here are the few steps:
Setting up a ticketfrei bot for your city is quite easy. Here are the
few steps:
First you need to install python3 and virtualenv with your favourite
package manager.
First you need to install python and virtualenv with your favourite package manager.
Create and activate virtualenv:
```shell
@ -34,19 +46,44 @@ Configure the bot:
cp config.toml.example config.toml
vim config.toml
```
Edit the account credentials, so your bot can use your accounts.
You can use a Twitter, a Mastodon, and Mail with the account. They
will communicate with each other; if someone warns others via Mail,
Twitter and Mastodon users will also see the message. And vice versa.
You have to configure all of the accounts via config.toml; it should
be fairly intuitive to enter the right values.
## Maintaining
There is one security hole: people could start mentioning the bot
with useless information, turning it into a spammer. That's why it
has to be maintained; if someone spams the bot, mute them and undo
the retweet. So it won't retweet their future tweets and the useless
retweet is deleted if someone tries to check if something was
retweeted in the last hour or something.
To this date, we have never heard of this happening though.
### blacklisting
Also add the words to the goodlist, which you want to require. A tweet is only retweeted, if it contains at least one of them. If you want to RT everything, just add your account name.
You also need to edit the goodlist and the blacklist. They are in the
"goodlists" and "blacklists" folders. All text files in those
directories will be used, so you should delete our templates; but
feel free to use them as an orientation.
There is also a blacklist, which you can use to automatically sort out malicious tweets. Be careful though, our filter can't read the intention with which a word was used. Maybe you wanted it there.
Just add the words to the goodlist, which you want to require. A
report is only spread, if it contains at least one of them. If you
want to RT everything, just add a ```*```.
Note that atm the good- & blacklist are still outside of config.toml, in separate files. we will repare this soon.
There is also a blacklist, which you can use to automatically sort
out malicious tweets. Be careful though, our filter can't read the
intention with which a word was used. Maybe you wanted it there.
### screen
To keep the bots running when you are logged out of the shell, you can use screen:
To keep the bots running when you are logged out of the shell, you
can use screen:
```shell
sudo apt-get install screen
@ -57,28 +94,3 @@ python3 ticketfrei.py
To log out of the screen session, press "ctrl+a", and then "d".
## ideas
* You can only use the twitter API if you have confirmed a phone number and sacrificed a penguin in a blood ritual. So we should build it in a way that it uses the twitter web GUI. It's difficult, but maybe it works. We had another twitter bot that worked similarly, years ago: https://github.com/b3yond/twitter-bot
* Build a tool that deletes wrong toots/tweets on both platforms, would work nicely with a web UI.
* write the muted people to the db, to easily undo the mutes if necessary.
## to do
Desktop/pycharm-community-2017.1.4/bin/pycharm.sh
- [x] Twitter: Crawl mentions
- [x] Mastodon: Crawl mentions
- [ ] Write toots/tweets to database/log
- [x] Twitter: retweet people
- [x] Mastodon: boost people
- [x] Twitter: access the API
- [ ] Web UI that lets you easily delete toots/tweets per db id and/or mute the tweet author
- [x] Write Bots as Classes to be easier implemented
- [x] Create extra Class for the filter
- [x] Put as much as possible into config.toml
- [x] Make both bots run on their own *and* next to each other
- [x] implement trigger class in retootbot
- [x] read config in retweetbot
- [x] put shutdown contact in config.toml
- [x] document what you have to configure if you setup the bot in another city
- [ ] write a script to setup the bot easily. ask the admin for the necessary information
- [ ] ongoing: solve issues

7
config.toml.example
View file

@ -3,6 +3,7 @@
name = 'yourcity_ticketfrei' # What you want the app to be called
[muser]
enabled = 'false' # set to true if you want to use Mastodon
email = 'youremail@server.tld' # E-mail address of your Mastodon account
password = 'yourpassword' # Password of your Mastodon account
server = 'yourmastodoninstance' # Instance where you have your Mastodon account
@ -14,12 +15,14 @@ consumer_key = "your_consumer_key"
consumer_secret = "your_consumer_secret"
[tuser]
enabled = 'false' # set to true if you want to use Twitter
# You get those keys when you follow these steps:
# https://developer.twitter.com/en/docs/basics/authentication/guides/access-tokens
access_token_key = "your_access_token_key"
access_token_secret = "your_acces_token_secret"
[mail]
enabled = 'false' # set to true if you want to use Mail notifications
# This is the mail the bot uses to send emails.
mailserver = "smtp.riseup.net"
user = "ticketfrei"
@ -28,14 +31,14 @@ passphrase = "sup3rs3cur3"
# when it breaks down), you should specify a contact email address:
#contact = "your_mail@riseup.net"
# Mailing list where you want to send warnings to
#list = "nbg_ticketfrei@lists.links-tech.org"
list = "yourcity_ticketfrei@lists.links-tech.org"
[web]
secret = "adoijf83wuc2mwipje8r"
[logging]
# The directory where logs should be stored.
logpath = "logs"
logpath = "logs/ticketfrei.log"
# [trigger]
# goodlists are one regex per line.

94
mailbot.py
View file

@ -9,6 +9,7 @@ import email
import logging
import pytoml as toml
import imaplib
import report
logger = logging.getLogger(__name__)
@ -19,7 +20,7 @@ class Mailbot(object):
other bots that it received mails.
"""
def __init__(self, config, trigger, history_path="last_mail"):
def __init__(self, config, history_path="last_mail"):
"""
Creates a Bot who listens to mails and forwards them to other
bots.
@ -27,7 +28,6 @@ class Mailbot(object):
:param config: (dictionary) config.toml as a dictionary of dictionaries
"""
self.config = config
self.trigger = trigger
self.history_path = history_path
self.last_mail = self.get_history(self.history_path)
@ -55,10 +55,20 @@ class Mailbot(object):
except:
logger.error('Mail sending failed', exc_info=True)
def listen(self):
def repost(self, status):
"""
listen for mails which contain goodwords but no badwords.
:return:
E-Mails don't have to be reposted - they already reached everyone on the mailing list.
The function still needs to be here because ticketfrei.py assumes it, and take the
report object they want to give us.
:param status: (report.Report object)
"""
pass
def crawl(self):
"""
crawl for new mails.
:return: msgs: (list of report.Report objects)
"""
rv, data = self.mailbox.select("Inbox")
msgs = []
@ -74,18 +84,21 @@ class Mailbot(object):
return msgs
msg = email.message_from_bytes(data[0][1])
# get a comparable date out of the email
date_tuple = email.utils.parsedate_tz(msg['Date'])
date_tuple = datetime.datetime.fromtimestamp(email.utils.mktime_tz(date_tuple))
date = (date_tuple - datetime.datetime(1970, 1, 1)).total_seconds()
if date > self.get_history(self.history_path):
self.last_mail = date
self.save_last_mail()
msgs.append(msg)
if not self.config['mail']['user'] + "@" + \
self.config["mail"]["mailserver"].partition(".")[2] in msg['From']:
# get a comparable date out of the email
date_tuple = email.utils.parsedate_tz(msg['Date'])
date_tuple = datetime.datetime.fromtimestamp(email.utils.mktime_tz(date_tuple))
date = (date_tuple - datetime.datetime(1970, 1, 1)).total_seconds()
if date > self.get_history(self.history_path):
self.last_mail = date
self.save_last()
msgs.append(self.make_report(msg))
return msgs
def get_history(self, path):
""" This counter is needed to keep track of your mails, so you
"""
This counter is needed to keep track of your mails, so you
don't double parse them
:param path: string: contains path to the file where the ID of the
@ -101,26 +114,26 @@ class Mailbot(object):
f.write(last_mail)
return float(last_mail)
def save_last_mail(self):
def save_last(self):
""" Saves the last retweeted tweet in last_mention. """
with open(self.history_path, "w") as f:
f.write(str(self.last_mail))
def send_report(self, statuses):
def post(self, status):
"""
sends reports by twitter & mastodon to a mailing list.
sends reports by other sources to a mailing list.
:param statuses: (list) of status strings
:param status: (report.Report object)
"""
for status in statuses:
mailer = sendmail.Mailer(self.config)
mailer.send(status, self.mailinglist, "Warnung: Kontrolleure gesehen")
mailer = sendmail.Mailer(self.config)
mailer.send(status.format(), self.mailinglist, "Warnung: Kontrolleure gesehen")
def to_social(self, msg):
def make_report(self, msg):
"""
sends a report from the mailing list to social
generates a report out of a mail
:param msg: email.parser.Message object
:return: post: (string) of author + text
:return: post: report.Report object
"""
# get a comparable date out of the email
date_tuple = email.utils.parsedate_tz(msg['Date'])
@ -131,26 +144,27 @@ class Mailbot(object):
# :todo take only the part before the @
text = msg.get_payload()
post = author + ": " + text
post = report.Report(author, "mail", text, None, date)
self.last_mail = date
self.save_last_mail()
self.save_last()
return post
def flow(self, statuses):
def flow(self, trigger, statuses):
"""
to be iterated
to be iterated. uses trigger to separate the sheep from the goats
:param statuses: (list) of statuses to send to mailinglist
:return: list of statuses to post in mastodon & twitter
:param statuses: (list of report.Report objects)
:return: statuses: (list of report.Report objects)
"""
self.send_report(statuses)
for status in statuses:
self.post(status)
msgs = self.listen()
msgs = self.crawl()
statuses = []
for msg in msgs:
if self.trigger.is_ok(msg.get_payload()):
statuses.append(self.to_social(msg))
if trigger.is_ok(msg.get_payload()):
statuses.append(msg)
return statuses
@ -159,22 +173,28 @@ if __name__ == "__main__":
with open('config.toml') as configfile:
config = toml.load(configfile)
# set log file
logger = logging.getLogger()
fh = logging.FileHandler(config['logging']['logpath'])
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
# initialise trigger
trigger = trigger.Trigger(config)
m = Mailbot(config, trigger)
# initialise mail bot
m = Mailbot(config)
statuses = []
try:
while 1:
print("Received Reports: " + str(m.flow(statuses)))
print("Received Reports: " + str(m.flow(trigger, statuses)))
time.sleep(1)
except KeyboardInterrupt:
print("Good bye. Remember to restart the bot!")
except:
logger.error('Shutdown', exc_info=True)
m.save_last_mail()
m.save_last()
try:
mailer = sendmail.Mailer(config)
mailer.send('', config['mail']['contact'],

4
promotion/flyer_polite.md
View file

@ -96,6 +96,10 @@ SCHWARZFAHREN erwischt wirst,
DANN nur weil Du noch nicht den Ticketfrei-Service deiner
VGN nutzt.
WENN
ÜBERWACHUNG sich lohnt,
DANN weil die
VAG öffentlich ist
# Eine Zukunft ohne Ticketautomaten und Kontrolleure

36
report.py Normal file
View file

@ -0,0 +1,36 @@
#!/usr/bin/env python3
class Report(object):
"""
A ticketfrei report object.
Toots, Tweets, and E-Mails can be formed into ticketfrei reports.
"""
def __init__(self, author, source, text, id, timestamp):
"""
Constructor of a ticketfrei report
:param author: username of the author
:param source: mastodon, twitter, or email
:param text: the text of the report
:param id: id in the network
:param timestamp: time of the report
"""
self.author = author
self.type = source
self.text = text
self.timestamp = timestamp
self.id = id
def format(self):
"""
Format the report for bot.post()
:rtype: string
:return: toot: text to be tooted, e.g. "_b3yond: There are
uniformed controllers in the U2 at Opernhaus."
"""
strng = self.author + ": " + self.text
return strng

98
retootbot.py
View file

@ -9,14 +9,14 @@ import time
import trigger
import logging
import sendmail
import report
logger = logging.getLogger(__name__)
class RetootBot(object):
def __init__(self, config, trigger):
def __init__(self, config):
self.config = config
self.trigger = trigger
self.client_id = self.register()
self.m = self.login()
@ -53,34 +53,76 @@ class RetootBot(object):
)
return m
def retoot(self, toots=()):
def save_last(self):
""" save the last seen toot """
try:
with os.fdopen(os.open('seen_toots.pickle.part', os.O_WRONLY | os.O_EXCL | os.O_CREAT), 'wb') as f:
pickle.dump(self.seen_toots, f)
except FileExistsError:
os.unlink('seen_toots.pickle.part')
with os.fdopen(os.open('seen_toots.pickle.part', os.O_WRONLY | os.O_EXCL | os.O_CREAT), 'wb') as f:
pickle.dump(self.seen_toots, f)
os.rename('seen_toots.pickle.part', 'seen_toots.pickle')
def crawl(self):
"""
Crawl mentions from Mastodon.
:return: list of statuses
"""
mentions = []
try:
all = self.m.notifications()
except: # mastodon.Mastodon.MastodonAPIError is unfortunately not in __init__.py
logger.error("Unknown Mastodon API Error.", exc_info=True)
return mentions
for status in all:
if (status['type'] == 'mention' and status['status']['id'] not in self.seen_toots):
# save state
self.seen_toots.add(status['status']['id'])
self.save_last()
# add mention to mentions
text = re.sub(r'<[^>]*>', '', status['status']['content'])
text = re.sub("(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)", "", text)
mentions.append(report.Report(status['account']['acct'],
"mastodon",
text,
status['status']['id'],
status['status']['created_at']))
return mentions
def repost(self, mention):
"""
Retoots a mention.
:param mention: (report.Report object)
"""
logger.info('Boosting toot from %s' % (
mention.format()))
self.m.status_reblog(mention.id)
def post(self, report):
"""
Toots a report from other sources.
:param report: (report.Report object)
"""
toot = report.format()
self.m.toot(toot)
def flow(self, trigger, reports=()):
# toot external provided messages
for toot in toots:
self.m.toot(toot)
for report in reports:
self.post(report)
# boost mentions
retoots = []
for notification in self.m.notifications():
if (notification['type'] == 'mention'
and notification['status']['id'] not in self.seen_toots):
self.seen_toots.add(notification['status']['id'])
text_content = re.sub(r'<[^>]*>', '',
notification['status']['content'])
if not self.trigger.is_ok(text_content):
continue
logger.info('Boosting toot from %s: %s' % (
notification['status']['account']['acct'],
notification['status']['content']))
self.m.status_reblog(notification['status']['id'])
retoots.append('%s: %s' % (
notification['status']['account']['acct'],
re.sub(r'@\S*', '', text_content)))
# If the Mastodon instance returns interesting Errors, add them here:
# save state
with os.fdopen(os.open('seen_toots.pickle.part', os.O_WRONLY | os.O_EXCL | os.O_CREAT), 'wb') as f:
pickle.dump(self.seen_toots, f)
os.rename('seen_toots.pickle.part', 'seen_toots.pickle')
for mention in self.crawl():
if not trigger.is_ok(mention.text):
continue
self.repost(mention)
retoots.append(mention)
# return mentions for mirroring
return retoots
@ -96,11 +138,11 @@ if __name__ == '__main__':
logger.addHandler(fh)
trigger = trigger.Trigger(config)
bot = RetootBot(config, trigger)
bot = RetootBot(config)
try:
while True:
bot.retoot()
bot.flow(trigger)
time.sleep(1)
except KeyboardInterrupt:
print("Good bye. Remember to restart the bot!")

101
retweetbot.py
View file

@ -1,10 +1,12 @@
#!/usr/bin/env python3
import tweepy
import re
import requests
import pytoml as toml
import trigger
from time import sleep
import report
import logging
import sendmail
@ -19,18 +21,14 @@ class RetweetBot(object):
api: The api object, generated with your oAuth keys, responsible for
communication with twitter rest API
triggers: a list of words, one of them has to be in a tweet for it to be
retweeted
last_mention: the ID of the last tweet which mentioned you
"""
def __init__(self, trigger, config, history_path="last_mention"):
def __init__(self, config, history_path="last_mention"):
"""
Initializes the bot and loads all the necessary data.
:param trigger: object of the trigger
:param config: (dictionary) config.toml as a dictionary of dictionaries
:param logger: object of the logger
:param history_path: Path to the file with ID of the last retweeted
Tweet
"""
@ -46,7 +44,6 @@ class RetweetBot(object):
self.history_path = history_path
self.last_mention = self.get_history(self.history_path)
self.trigger = trigger
self.waitcounter = 0
def get_api_keys(self):
@ -85,7 +82,7 @@ class RetweetBot(object):
f.write(last_mention)
return int(last_mention)
def save_last_mention(self):
def save_last(self):
""" Saves the last retweeted tweet in last_mention. """
with open(self.history_path, "w") as f:
f.write(str(self.last_mention))
@ -101,53 +98,53 @@ class RetweetBot(object):
self.waitcounter -= 1
return self.waitcounter
def format_mastodon(self, status):
"""
Bridge your Retweets to mastodon.
:rtype: string
:param status: Object of a tweet.
:return: toot: text tooted on mastodon, e.g. "_b3yond: There are
uniformed controllers in the U2 at Opernhaus."
"""
toot = status.user.name + ": " + status.text
return toot
def crawl_mentions(self):
def crawl(self):
"""
crawls all Tweets which mention the bot from the twitter rest API.
:return: list of Status objects
:return: reports: (list of report.Report objects)
"""
reports = []
try:
if not self.waiting():
if self.last_mention == 0:
mentions = self.api.mentions_timeline()
else:
mentions = self.api.mentions_timeline(since_id=self.last_mention)
return mentions
for status in mentions:
text = re.sub("(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)", "", status.text)
reports.append(report.Report(status.author.screen_name,
"twitter",
text,
status.id,
status.created_at))
self.save_last()
return reports
except tweepy.RateLimitError:
logger.error("Twitter API Error: Rate Limit Exceeded", exc_info=True)
self.waitcounter += 60*15 + 1
except requests.exceptions.ConnectionError:
logger.error("Twitter API Error: Bad Connection", exc_info=True)
self.waitcounter += 10
except tweepy.TweepError:
logger.error("Twitter API Error: General Error", exc_info=True)
return []
def retweet(self, status):
def repost(self, status):
"""
Retweets a given tweet.
:param status: A tweet object.
:param status: (report.Report object)
:return: toot: string of the tweet, to toot on mastodon.
"""
while 1:
try:
self.api.retweet(status.id)
logger.info("Retweeted: " + self.format_mastodon(status))
logger.info("Retweeted: " + status.format())
if status.id > self.last_mention:
self.last_mention = status.id
return self.format_mastodon(status)
self.save_last()
return status.format()
except requests.exceptions.ConnectionError:
logger.error("Twitter API Error: Bad Connection", exc_info=True)
sleep(10)
@ -156,75 +153,81 @@ class RetweetBot(object):
logger.error("Twitter Error", exc_info=True)
if status.id > self.last_mention:
self.last_mention = status.id
self.save_last()
return None
def tweet(self, post):
def post(self, status):
"""
Tweet a post.
:param post: String with the text to tweet.
:param status: (report.Report object)
"""
if len(post) > 280:
post = post[:280 - 4] + u' ...'
text = status.format()
if len(text) > 280:
text = status.text[:280 - 4] + u' ...'
while 1:
try:
self.api.update_status(status=post)
self.api.update_status(status=text)
return
except requests.exceptions.ConnectionError:
logger.error("Twitter API Error: Bad Connection", exc_info=True)
sleep(10)
def flow(self, to_tweet=()):
def flow(self, trigger, to_tweet=()):
""" The flow of crawling mentions and retweeting them.
:param to_tweet: list of strings to tweet
:return list of retweeted tweets, to toot on mastodon
"""
# Tweet the toots the Retootbot gives to us
# Tweet the reports from other sources
for post in to_tweet:
self.tweet(post)
self.post(post)
# Store all mentions in a list of Status Objects
mentions = self.crawl_mentions()
mastodon = []
mentions = self.crawl()
# initialise list of strings for other bots
all_tweets = []
for status in mentions:
# Is the Text of the Tweet in the triggerlist?
if self.trigger.is_ok(status.text):
if trigger.is_ok(status.text):
# Retweet status
toot = self.retweet(status)
toot = self.repost(status)
if toot:
mastodon.append(toot)
all_tweets.append(toot)
# save the id so it doesn't get crawled again
if status.id > self.last_mention:
self.last_mention = status.id
self.save_last_mention()
# Return Retweets for tooting on mastodon
return mastodon
# Return Retweets for posting on other bots
return all_tweets
if __name__ == "__main__":
# create an Api object
# get the config dict of dicts
with open('config.toml') as configfile:
config = toml.load(configfile)
# set log file
fh = logging.FileHandler(config['logging']['logpath'])
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
# initialise trigger
trigger = trigger.Trigger(config)
bot = RetweetBot(trigger, config)
# initialise twitter bot
bot = RetweetBot(config)
try:
while True:
bot.flow()
# :todo separate into small functions
bot.flow(trigger)
sleep(60)
except KeyboardInterrupt:
print("Good bye. Remember to restart the bot!")
except:
logger.error('Shutdown', exc_info=True)
bot.save_last_mention()
bot.save_last()
try:
mailer = sendmail.Mailer(config)
mailer.send('', config['mail']['contact'],

31
ticketfrei.py
View file

@ -7,34 +7,51 @@ import sendmail
from retootbot import RetootBot
from retweetbot import RetweetBot
from mailbot import Mailbot
from trigger import Trigger
if __name__ == '__main__':
# read config in TOML format (https://github.com/toml-lang/toml#toml)
with open('config.toml') as configfile:
config = toml.load(configfile)
# set log file
logger = logging.getLogger()
fh = logging.FileHandler(config['logging']['logpath'])
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
trigger = Trigger(config)
mbot = RetootBot(config, trigger)
tbot = RetweetBot(trigger, config)
bots = []
if config["muser"]["enabled"] != "false":
bots.append(RetootBot(config))
if config["tuser"]["enabled"] != "false":
bots.append(RetweetBot(config))
if config["mail"]["enabled"] != "false":
bots.append(Mailbot(config))
try:
statuses = []
while True:
statuses = mbot.retoot(statuses)
statuses = tbot.flow(statuses)
time.sleep(60)
for bot in bots:
reports = bot.crawl()
for status in reports:
if not trigger.is_ok(status.text):
continue
for bot2 in bots:
if bot == bot2:
bot2.repost(status)
else:
bot2.post(status)
time.sleep(60) # twitter rate limit >.<
except KeyboardInterrupt:
print("Good bye. Remember to restart the bot!")
except:
logger.error('Shutdown', exc_info=True)
tbot.save_last_mention()
for bot in bots:
bot.save_last()
try:
mailer = sendmail.Mailer(config)
mailer.send('', config['mail']['contact'],