merged ticketfrei 1.0 into multi-deployment #3
This commit is contained in:
commit
63cf134ffa
86
README.md
86
README.md
|
@ -1,21 +1,33 @@
|
||||||
# Ticketfrei micro messaging bot
|
# Ticketfrei micro messaging bot
|
||||||
|
|
||||||
|
Version: 1.0
|
||||||
|
|
||||||
<!-- This mastodon/twitter bot has one purpose - breaking the law. -->
|
<!-- This mastodon/twitter bot has one purpose - breaking the law. -->
|
||||||
|
|
||||||
The functionality is simple: it retweets every tweet where it is mentioned.
|
The functionality is simple: it retweets every tweet where it is
|
||||||
|
mentioned.
|
||||||
|
|
||||||
This leads to a community which evolves around it; if you see ticket controllers, you tweet their location and mention the bot. The bot then retweets your tweet and others can read the info and think twice if they want to buy a ticket. If enough people, a critical mass, participate for the bot to become reliable, you have positive self-reinforcing dynamics.
|
This leads to a community which evolves around it; if you see ticket
|
||||||
|
controllers, you tweet their location and mention the bot. The bot
|
||||||
|
then retweets your tweet and others can read the info and think twice
|
||||||
|
if they want to buy a ticket. If enough people, a critical mass,
|
||||||
|
participate for the bot to become reliable, you have positive
|
||||||
|
self-reinforcing dynamics.
|
||||||
|
|
||||||
There is one security hole: people could start mentioning the bot with useless information, turning it into a spammer. That's why it has to be maintained; if someone spams the bot, mute them and undo the retweet. So it won't retweet their future tweets and the useless retweet is deleted if someone tries to check if something was retweeted in the last hour or something.
|
In the promotion folder, you will find some promotion material you
|
||||||
To this date, we have never heard of this happening though.
|
can use to build up such a community in your city. It is in german
|
||||||
|
though =/
|
||||||
|
|
||||||
Website: https://wiki.links-it.de/IT/Ticketfrei
|
Website: https://wiki.links-tech.org/IT/Ticketfrei
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
Setting up a ticketfrei bot for your city is quite easy. Here are the few steps:
|
Setting up a ticketfrei bot for your city is quite easy. Here are the
|
||||||
|
few steps:
|
||||||
|
|
||||||
|
First you need to install python3 and virtualenv with your favourite
|
||||||
|
package manager.
|
||||||
|
|
||||||
First you need to install python and virtualenv with your favourite package manager.
|
|
||||||
Create and activate virtualenv:
|
Create and activate virtualenv:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
|
@ -34,19 +46,44 @@ Configure the bot:
|
||||||
cp config.toml.example config.toml
|
cp config.toml.example config.toml
|
||||||
vim config.toml
|
vim config.toml
|
||||||
```
|
```
|
||||||
Edit the account credentials, so your bot can use your accounts.
|
|
||||||
|
You can use a Twitter, a Mastodon, and Mail with the account. They
|
||||||
|
will communicate with each other; if someone warns others via Mail,
|
||||||
|
Twitter and Mastodon users will also see the message. And vice versa.
|
||||||
|
|
||||||
|
You have to configure all of the accounts via config.toml; it should
|
||||||
|
be fairly intuitive to enter the right values.
|
||||||
|
|
||||||
|
## Maintaining
|
||||||
|
|
||||||
|
There is one security hole: people could start mentioning the bot
|
||||||
|
with useless information, turning it into a spammer. That's why it
|
||||||
|
has to be maintained; if someone spams the bot, mute them and undo
|
||||||
|
the retweet. So it won't retweet their future tweets and the useless
|
||||||
|
retweet is deleted if someone tries to check if something was
|
||||||
|
retweeted in the last hour or something.
|
||||||
|
|
||||||
|
To this date, we have never heard of this happening though.
|
||||||
|
|
||||||
### blacklisting
|
### blacklisting
|
||||||
|
|
||||||
Also add the words to the goodlist, which you want to require. A tweet is only retweeted, if it contains at least one of them. If you want to RT everything, just add your account name.
|
You also need to edit the goodlist and the blacklist. They are in the
|
||||||
|
"goodlists" and "blacklists" folders. All text files in those
|
||||||
|
directories will be used, so you should delete our templates; but
|
||||||
|
feel free to use them as an orientation.
|
||||||
|
|
||||||
There is also a blacklist, which you can use to automatically sort out malicious tweets. Be careful though, our filter can't read the intention with which a word was used. Maybe you wanted it there.
|
Just add the words to the goodlist, which you want to require. A
|
||||||
|
report is only spread, if it contains at least one of them. If you
|
||||||
|
want to RT everything, just add a ```*```.
|
||||||
|
|
||||||
Note that atm the good- & blacklist are still outside of config.toml, in separate files. we will repare this soon.
|
There is also a blacklist, which you can use to automatically sort
|
||||||
|
out malicious tweets. Be careful though, our filter can't read the
|
||||||
|
intention with which a word was used. Maybe you wanted it there.
|
||||||
|
|
||||||
### screen
|
### screen
|
||||||
|
|
||||||
To keep the bots running when you are logged out of the shell, you can use screen:
|
To keep the bots running when you are logged out of the shell, you
|
||||||
|
can use screen:
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
sudo apt-get install screen
|
sudo apt-get install screen
|
||||||
|
@ -57,28 +94,3 @@ python3 ticketfrei.py
|
||||||
|
|
||||||
To log out of the screen session, press "ctrl+a", and then "d".
|
To log out of the screen session, press "ctrl+a", and then "d".
|
||||||
|
|
||||||
## ideas
|
|
||||||
|
|
||||||
* You can only use the twitter API if you have confirmed a phone number and sacrificed a penguin in a blood ritual. So we should build it in a way that it uses the twitter web GUI. It's difficult, but maybe it works. We had another twitter bot that worked similarly, years ago: https://github.com/b3yond/twitter-bot
|
|
||||||
* Build a tool that deletes wrong toots/tweets on both platforms, would work nicely with a web UI.
|
|
||||||
* write the muted people to the db, to easily undo the mutes if necessary.
|
|
||||||
|
|
||||||
## to do
|
|
||||||
Desktop/pycharm-community-2017.1.4/bin/pycharm.sh
|
|
||||||
- [x] Twitter: Crawl mentions
|
|
||||||
- [x] Mastodon: Crawl mentions
|
|
||||||
- [ ] Write toots/tweets to database/log
|
|
||||||
- [x] Twitter: retweet people
|
|
||||||
- [x] Mastodon: boost people
|
|
||||||
- [x] Twitter: access the API
|
|
||||||
- [ ] Web UI that lets you easily delete toots/tweets per db id and/or mute the tweet author
|
|
||||||
- [x] Write Bots as Classes to be easier implemented
|
|
||||||
- [x] Create extra Class for the filter
|
|
||||||
- [x] Put as much as possible into config.toml
|
|
||||||
- [x] Make both bots run on their own *and* next to each other
|
|
||||||
- [x] implement trigger class in retootbot
|
|
||||||
- [x] read config in retweetbot
|
|
||||||
- [x] put shutdown contact in config.toml
|
|
||||||
- [x] document what you have to configure if you setup the bot in another city
|
|
||||||
- [ ] write a script to setup the bot easily. ask the admin for the necessary information
|
|
||||||
- [ ] ongoing: solve issues
|
|
||||||
|
|
|
@ -3,6 +3,7 @@
|
||||||
name = 'yourcity_ticketfrei' # What you want the app to be called
|
name = 'yourcity_ticketfrei' # What you want the app to be called
|
||||||
|
|
||||||
[muser]
|
[muser]
|
||||||
|
enabled = 'false' # set to true if you want to use Mastodon
|
||||||
email = 'youremail@server.tld' # E-mail address of your Mastodon account
|
email = 'youremail@server.tld' # E-mail address of your Mastodon account
|
||||||
password = 'yourpassword' # Password of your Mastodon account
|
password = 'yourpassword' # Password of your Mastodon account
|
||||||
server = 'yourmastodoninstance' # Instance where you have your Mastodon account
|
server = 'yourmastodoninstance' # Instance where you have your Mastodon account
|
||||||
|
@ -14,12 +15,14 @@ consumer_key = "your_consumer_key"
|
||||||
consumer_secret = "your_consumer_secret"
|
consumer_secret = "your_consumer_secret"
|
||||||
|
|
||||||
[tuser]
|
[tuser]
|
||||||
|
enabled = 'false' # set to true if you want to use Twitter
|
||||||
# You get those keys when you follow these steps:
|
# You get those keys when you follow these steps:
|
||||||
# https://developer.twitter.com/en/docs/basics/authentication/guides/access-tokens
|
# https://developer.twitter.com/en/docs/basics/authentication/guides/access-tokens
|
||||||
access_token_key = "your_access_token_key"
|
access_token_key = "your_access_token_key"
|
||||||
access_token_secret = "your_acces_token_secret"
|
access_token_secret = "your_acces_token_secret"
|
||||||
|
|
||||||
[mail]
|
[mail]
|
||||||
|
enabled = 'false' # set to true if you want to use Mail notifications
|
||||||
# This is the mail the bot uses to send emails.
|
# This is the mail the bot uses to send emails.
|
||||||
mailserver = "smtp.riseup.net"
|
mailserver = "smtp.riseup.net"
|
||||||
user = "ticketfrei"
|
user = "ticketfrei"
|
||||||
|
@ -28,14 +31,14 @@ passphrase = "sup3rs3cur3"
|
||||||
# when it breaks down), you should specify a contact email address:
|
# when it breaks down), you should specify a contact email address:
|
||||||
#contact = "your_mail@riseup.net"
|
#contact = "your_mail@riseup.net"
|
||||||
# Mailing list where you want to send warnings to
|
# Mailing list where you want to send warnings to
|
||||||
#list = "nbg_ticketfrei@lists.links-tech.org"
|
list = "yourcity_ticketfrei@lists.links-tech.org"
|
||||||
|
|
||||||
[web]
|
[web]
|
||||||
secret = "adoijf83wuc2mwipje8r"
|
secret = "adoijf83wuc2mwipje8r"
|
||||||
|
|
||||||
[logging]
|
[logging]
|
||||||
# The directory where logs should be stored.
|
# The directory where logs should be stored.
|
||||||
logpath = "logs"
|
logpath = "logs/ticketfrei.log"
|
||||||
|
|
||||||
# [trigger]
|
# [trigger]
|
||||||
# goodlists are one regex per line.
|
# goodlists are one regex per line.
|
||||||
|
|
80
mailbot.py
80
mailbot.py
|
@ -9,6 +9,7 @@ import email
|
||||||
import logging
|
import logging
|
||||||
import pytoml as toml
|
import pytoml as toml
|
||||||
import imaplib
|
import imaplib
|
||||||
|
import report
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@ -19,7 +20,7 @@ class Mailbot(object):
|
||||||
other bots that it received mails.
|
other bots that it received mails.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, config, trigger, history_path="last_mail"):
|
def __init__(self, config, history_path="last_mail"):
|
||||||
"""
|
"""
|
||||||
Creates a Bot who listens to mails and forwards them to other
|
Creates a Bot who listens to mails and forwards them to other
|
||||||
bots.
|
bots.
|
||||||
|
@ -27,7 +28,6 @@ class Mailbot(object):
|
||||||
:param config: (dictionary) config.toml as a dictionary of dictionaries
|
:param config: (dictionary) config.toml as a dictionary of dictionaries
|
||||||
"""
|
"""
|
||||||
self.config = config
|
self.config = config
|
||||||
self.trigger = trigger
|
|
||||||
|
|
||||||
self.history_path = history_path
|
self.history_path = history_path
|
||||||
self.last_mail = self.get_history(self.history_path)
|
self.last_mail = self.get_history(self.history_path)
|
||||||
|
@ -55,10 +55,20 @@ class Mailbot(object):
|
||||||
except:
|
except:
|
||||||
logger.error('Mail sending failed', exc_info=True)
|
logger.error('Mail sending failed', exc_info=True)
|
||||||
|
|
||||||
def listen(self):
|
def repost(self, status):
|
||||||
"""
|
"""
|
||||||
listen for mails which contain goodwords but no badwords.
|
E-Mails don't have to be reposted - they already reached everyone on the mailing list.
|
||||||
:return:
|
The function still needs to be here because ticketfrei.py assumes it, and take the
|
||||||
|
report object they want to give us.
|
||||||
|
|
||||||
|
:param status: (report.Report object)
|
||||||
|
"""
|
||||||
|
pass
|
||||||
|
|
||||||
|
def crawl(self):
|
||||||
|
"""
|
||||||
|
crawl for new mails.
|
||||||
|
:return: msgs: (list of report.Report objects)
|
||||||
"""
|
"""
|
||||||
rv, data = self.mailbox.select("Inbox")
|
rv, data = self.mailbox.select("Inbox")
|
||||||
msgs = []
|
msgs = []
|
||||||
|
@ -74,18 +84,21 @@ class Mailbot(object):
|
||||||
return msgs
|
return msgs
|
||||||
msg = email.message_from_bytes(data[0][1])
|
msg = email.message_from_bytes(data[0][1])
|
||||||
|
|
||||||
|
if not self.config['mail']['user'] + "@" + \
|
||||||
|
self.config["mail"]["mailserver"].partition(".")[2] in msg['From']:
|
||||||
# get a comparable date out of the email
|
# get a comparable date out of the email
|
||||||
date_tuple = email.utils.parsedate_tz(msg['Date'])
|
date_tuple = email.utils.parsedate_tz(msg['Date'])
|
||||||
date_tuple = datetime.datetime.fromtimestamp(email.utils.mktime_tz(date_tuple))
|
date_tuple = datetime.datetime.fromtimestamp(email.utils.mktime_tz(date_tuple))
|
||||||
date = (date_tuple - datetime.datetime(1970, 1, 1)).total_seconds()
|
date = (date_tuple - datetime.datetime(1970, 1, 1)).total_seconds()
|
||||||
if date > self.get_history(self.history_path):
|
if date > self.get_history(self.history_path):
|
||||||
self.last_mail = date
|
self.last_mail = date
|
||||||
self.save_last_mail()
|
self.save_last()
|
||||||
msgs.append(msg)
|
msgs.append(self.make_report(msg))
|
||||||
return msgs
|
return msgs
|
||||||
|
|
||||||
def get_history(self, path):
|
def get_history(self, path):
|
||||||
""" This counter is needed to keep track of your mails, so you
|
"""
|
||||||
|
This counter is needed to keep track of your mails, so you
|
||||||
don't double parse them
|
don't double parse them
|
||||||
|
|
||||||
:param path: string: contains path to the file where the ID of the
|
:param path: string: contains path to the file where the ID of the
|
||||||
|
@ -101,26 +114,26 @@ class Mailbot(object):
|
||||||
f.write(last_mail)
|
f.write(last_mail)
|
||||||
return float(last_mail)
|
return float(last_mail)
|
||||||
|
|
||||||
def save_last_mail(self):
|
def save_last(self):
|
||||||
""" Saves the last retweeted tweet in last_mention. """
|
""" Saves the last retweeted tweet in last_mention. """
|
||||||
with open(self.history_path, "w") as f:
|
with open(self.history_path, "w") as f:
|
||||||
f.write(str(self.last_mail))
|
f.write(str(self.last_mail))
|
||||||
|
|
||||||
def send_report(self, statuses):
|
def post(self, status):
|
||||||
"""
|
"""
|
||||||
sends reports by twitter & mastodon to a mailing list.
|
sends reports by other sources to a mailing list.
|
||||||
|
|
||||||
:param statuses: (list) of status strings
|
:param status: (report.Report object)
|
||||||
"""
|
"""
|
||||||
for status in statuses:
|
|
||||||
mailer = sendmail.Mailer(self.config)
|
mailer = sendmail.Mailer(self.config)
|
||||||
mailer.send(status, self.mailinglist, "Warnung: Kontrolleure gesehen")
|
mailer.send(status.format(), self.mailinglist, "Warnung: Kontrolleure gesehen")
|
||||||
|
|
||||||
def to_social(self, msg):
|
def make_report(self, msg):
|
||||||
"""
|
"""
|
||||||
sends a report from the mailing list to social
|
generates a report out of a mail
|
||||||
|
|
||||||
:param msg: email.parser.Message object
|
:param msg: email.parser.Message object
|
||||||
:return: post: (string) of author + text
|
:return: post: report.Report object
|
||||||
"""
|
"""
|
||||||
# get a comparable date out of the email
|
# get a comparable date out of the email
|
||||||
date_tuple = email.utils.parsedate_tz(msg['Date'])
|
date_tuple = email.utils.parsedate_tz(msg['Date'])
|
||||||
|
@ -131,26 +144,27 @@ class Mailbot(object):
|
||||||
# :todo take only the part before the @
|
# :todo take only the part before the @
|
||||||
|
|
||||||
text = msg.get_payload()
|
text = msg.get_payload()
|
||||||
post = author + ": " + text
|
post = report.Report(author, "mail", text, None, date)
|
||||||
self.last_mail = date
|
self.last_mail = date
|
||||||
self.save_last_mail()
|
self.save_last()
|
||||||
return post
|
return post
|
||||||
|
|
||||||
def flow(self, statuses):
|
def flow(self, trigger, statuses):
|
||||||
"""
|
"""
|
||||||
to be iterated
|
to be iterated. uses trigger to separate the sheep from the goats
|
||||||
|
|
||||||
:param statuses: (list) of statuses to send to mailinglist
|
:param statuses: (list of report.Report objects)
|
||||||
:return: list of statuses to post in mastodon & twitter
|
:return: statuses: (list of report.Report objects)
|
||||||
"""
|
"""
|
||||||
self.send_report(statuses)
|
for status in statuses:
|
||||||
|
self.post(status)
|
||||||
|
|
||||||
msgs = self.listen()
|
msgs = self.crawl()
|
||||||
|
|
||||||
statuses = []
|
statuses = []
|
||||||
for msg in msgs:
|
for msg in msgs:
|
||||||
if self.trigger.is_ok(msg.get_payload()):
|
if trigger.is_ok(msg.get_payload()):
|
||||||
statuses.append(self.to_social(msg))
|
statuses.append(msg)
|
||||||
return statuses
|
return statuses
|
||||||
|
|
||||||
|
|
||||||
|
@ -159,22 +173,28 @@ if __name__ == "__main__":
|
||||||
with open('config.toml') as configfile:
|
with open('config.toml') as configfile:
|
||||||
config = toml.load(configfile)
|
config = toml.load(configfile)
|
||||||
|
|
||||||
|
# set log file
|
||||||
|
logger = logging.getLogger()
|
||||||
fh = logging.FileHandler(config['logging']['logpath'])
|
fh = logging.FileHandler(config['logging']['logpath'])
|
||||||
fh.setLevel(logging.DEBUG)
|
fh.setLevel(logging.DEBUG)
|
||||||
logger.addHandler(fh)
|
logger.addHandler(fh)
|
||||||
|
|
||||||
|
# initialise trigger
|
||||||
trigger = trigger.Trigger(config)
|
trigger = trigger.Trigger(config)
|
||||||
m = Mailbot(config, trigger)
|
|
||||||
|
# initialise mail bot
|
||||||
|
m = Mailbot(config)
|
||||||
|
|
||||||
statuses = []
|
statuses = []
|
||||||
try:
|
try:
|
||||||
while 1:
|
while 1:
|
||||||
print("Received Reports: " + str(m.flow(statuses)))
|
print("Received Reports: " + str(m.flow(trigger, statuses)))
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
print("Good bye. Remember to restart the bot!")
|
print("Good bye. Remember to restart the bot!")
|
||||||
except:
|
except:
|
||||||
logger.error('Shutdown', exc_info=True)
|
logger.error('Shutdown', exc_info=True)
|
||||||
m.save_last_mail()
|
m.save_last()
|
||||||
try:
|
try:
|
||||||
mailer = sendmail.Mailer(config)
|
mailer = sendmail.Mailer(config)
|
||||||
mailer.send('', config['mail']['contact'],
|
mailer.send('', config['mail']['contact'],
|
||||||
|
|
|
@ -96,6 +96,10 @@ SCHWARZFAHREN erwischt wirst,
|
||||||
DANN nur weil Du noch nicht den Ticketfrei-Service deiner
|
DANN nur weil Du noch nicht den Ticketfrei-Service deiner
|
||||||
VGN nutzt.
|
VGN nutzt.
|
||||||
|
|
||||||
|
WENN
|
||||||
|
ÜBERWACHUNG sich lohnt,
|
||||||
|
DANN weil die
|
||||||
|
VAG öffentlich ist
|
||||||
|
|
||||||
# Eine Zukunft ohne Ticketautomaten und Kontrolleure
|
# Eine Zukunft ohne Ticketautomaten und Kontrolleure
|
||||||
|
|
||||||
|
|
36
report.py
Normal file
36
report.py
Normal file
|
@ -0,0 +1,36 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
class Report(object):
|
||||||
|
"""
|
||||||
|
A ticketfrei report object.
|
||||||
|
|
||||||
|
Toots, Tweets, and E-Mails can be formed into ticketfrei reports.
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, author, source, text, id, timestamp):
|
||||||
|
"""
|
||||||
|
Constructor of a ticketfrei report
|
||||||
|
|
||||||
|
:param author: username of the author
|
||||||
|
:param source: mastodon, twitter, or email
|
||||||
|
:param text: the text of the report
|
||||||
|
:param id: id in the network
|
||||||
|
:param timestamp: time of the report
|
||||||
|
"""
|
||||||
|
self.author = author
|
||||||
|
self.type = source
|
||||||
|
self.text = text
|
||||||
|
self.timestamp = timestamp
|
||||||
|
self.id = id
|
||||||
|
|
||||||
|
def format(self):
|
||||||
|
"""
|
||||||
|
Format the report for bot.post()
|
||||||
|
|
||||||
|
:rtype: string
|
||||||
|
:return: toot: text to be tooted, e.g. "_b3yond: There are
|
||||||
|
uniformed controllers in the U2 at Opernhaus."
|
||||||
|
"""
|
||||||
|
strng = self.author + ": " + self.text
|
||||||
|
return strng
|
100
retootbot.py
100
retootbot.py
|
@ -9,14 +9,14 @@ import time
|
||||||
import trigger
|
import trigger
|
||||||
import logging
|
import logging
|
||||||
import sendmail
|
import sendmail
|
||||||
|
import report
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
class RetootBot(object):
|
class RetootBot(object):
|
||||||
def __init__(self, config, trigger):
|
def __init__(self, config):
|
||||||
self.config = config
|
self.config = config
|
||||||
self.trigger = trigger
|
|
||||||
self.client_id = self.register()
|
self.client_id = self.register()
|
||||||
self.m = self.login()
|
self.m = self.login()
|
||||||
|
|
||||||
|
@ -53,35 +53,77 @@ class RetootBot(object):
|
||||||
)
|
)
|
||||||
return m
|
return m
|
||||||
|
|
||||||
def retoot(self, toots=()):
|
def save_last(self):
|
||||||
# toot external provided messages
|
""" save the last seen toot """
|
||||||
for toot in toots:
|
try:
|
||||||
self.m.toot(toot)
|
with os.fdopen(os.open('seen_toots.pickle.part', os.O_WRONLY | os.O_EXCL | os.O_CREAT), 'wb') as f:
|
||||||
|
pickle.dump(self.seen_toots, f)
|
||||||
# boost mentions
|
except FileExistsError:
|
||||||
retoots = []
|
os.unlink('seen_toots.pickle.part')
|
||||||
for notification in self.m.notifications():
|
|
||||||
if (notification['type'] == 'mention'
|
|
||||||
and notification['status']['id'] not in self.seen_toots):
|
|
||||||
self.seen_toots.add(notification['status']['id'])
|
|
||||||
text_content = re.sub(r'<[^>]*>', '',
|
|
||||||
notification['status']['content'])
|
|
||||||
if not self.trigger.is_ok(text_content):
|
|
||||||
continue
|
|
||||||
logger.info('Boosting toot from %s: %s' % (
|
|
||||||
notification['status']['account']['acct'],
|
|
||||||
notification['status']['content']))
|
|
||||||
self.m.status_reblog(notification['status']['id'])
|
|
||||||
retoots.append('%s: %s' % (
|
|
||||||
notification['status']['account']['acct'],
|
|
||||||
re.sub(r'@\S*', '', text_content)))
|
|
||||||
# If the Mastodon instance returns interesting Errors, add them here:
|
|
||||||
|
|
||||||
# save state
|
|
||||||
with os.fdopen(os.open('seen_toots.pickle.part', os.O_WRONLY | os.O_EXCL | os.O_CREAT), 'wb') as f:
|
with os.fdopen(os.open('seen_toots.pickle.part', os.O_WRONLY | os.O_EXCL | os.O_CREAT), 'wb') as f:
|
||||||
pickle.dump(self.seen_toots, f)
|
pickle.dump(self.seen_toots, f)
|
||||||
os.rename('seen_toots.pickle.part', 'seen_toots.pickle')
|
os.rename('seen_toots.pickle.part', 'seen_toots.pickle')
|
||||||
|
|
||||||
|
def crawl(self):
|
||||||
|
"""
|
||||||
|
Crawl mentions from Mastodon.
|
||||||
|
|
||||||
|
:return: list of statuses
|
||||||
|
"""
|
||||||
|
mentions = []
|
||||||
|
try:
|
||||||
|
all = self.m.notifications()
|
||||||
|
except: # mastodon.Mastodon.MastodonAPIError is unfortunately not in __init__.py
|
||||||
|
logger.error("Unknown Mastodon API Error.", exc_info=True)
|
||||||
|
return mentions
|
||||||
|
for status in all:
|
||||||
|
if (status['type'] == 'mention' and status['status']['id'] not in self.seen_toots):
|
||||||
|
# save state
|
||||||
|
self.seen_toots.add(status['status']['id'])
|
||||||
|
self.save_last()
|
||||||
|
# add mention to mentions
|
||||||
|
text = re.sub(r'<[^>]*>', '', status['status']['content'])
|
||||||
|
text = re.sub("(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)", "", text)
|
||||||
|
mentions.append(report.Report(status['account']['acct'],
|
||||||
|
"mastodon",
|
||||||
|
text,
|
||||||
|
status['status']['id'],
|
||||||
|
status['status']['created_at']))
|
||||||
|
return mentions
|
||||||
|
|
||||||
|
def repost(self, mention):
|
||||||
|
"""
|
||||||
|
Retoots a mention.
|
||||||
|
|
||||||
|
:param mention: (report.Report object)
|
||||||
|
"""
|
||||||
|
logger.info('Boosting toot from %s' % (
|
||||||
|
mention.format()))
|
||||||
|
self.m.status_reblog(mention.id)
|
||||||
|
|
||||||
|
|
||||||
|
def post(self, report):
|
||||||
|
"""
|
||||||
|
Toots a report from other sources.
|
||||||
|
|
||||||
|
:param report: (report.Report object)
|
||||||
|
"""
|
||||||
|
toot = report.format()
|
||||||
|
self.m.toot(toot)
|
||||||
|
|
||||||
|
def flow(self, trigger, reports=()):
|
||||||
|
# toot external provided messages
|
||||||
|
for report in reports:
|
||||||
|
self.post(report)
|
||||||
|
|
||||||
|
# boost mentions
|
||||||
|
retoots = []
|
||||||
|
for mention in self.crawl():
|
||||||
|
if not trigger.is_ok(mention.text):
|
||||||
|
continue
|
||||||
|
self.repost(mention)
|
||||||
|
retoots.append(mention)
|
||||||
|
|
||||||
# return mentions for mirroring
|
# return mentions for mirroring
|
||||||
return retoots
|
return retoots
|
||||||
|
|
||||||
|
@ -96,11 +138,11 @@ if __name__ == '__main__':
|
||||||
logger.addHandler(fh)
|
logger.addHandler(fh)
|
||||||
|
|
||||||
trigger = trigger.Trigger(config)
|
trigger = trigger.Trigger(config)
|
||||||
bot = RetootBot(config, trigger)
|
bot = RetootBot(config)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
while True:
|
while True:
|
||||||
bot.retoot()
|
bot.flow(trigger)
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
print("Good bye. Remember to restart the bot!")
|
print("Good bye. Remember to restart the bot!")
|
||||||
|
|
101
retweetbot.py
101
retweetbot.py
|
@ -1,10 +1,12 @@
|
||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
import tweepy
|
import tweepy
|
||||||
|
import re
|
||||||
import requests
|
import requests
|
||||||
import pytoml as toml
|
import pytoml as toml
|
||||||
import trigger
|
import trigger
|
||||||
from time import sleep
|
from time import sleep
|
||||||
|
import report
|
||||||
import logging
|
import logging
|
||||||
import sendmail
|
import sendmail
|
||||||
|
|
||||||
|
@ -19,18 +21,14 @@ class RetweetBot(object):
|
||||||
|
|
||||||
api: The api object, generated with your oAuth keys, responsible for
|
api: The api object, generated with your oAuth keys, responsible for
|
||||||
communication with twitter rest API
|
communication with twitter rest API
|
||||||
triggers: a list of words, one of them has to be in a tweet for it to be
|
|
||||||
retweeted
|
|
||||||
last_mention: the ID of the last tweet which mentioned you
|
last_mention: the ID of the last tweet which mentioned you
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, trigger, config, history_path="last_mention"):
|
def __init__(self, config, history_path="last_mention"):
|
||||||
"""
|
"""
|
||||||
Initializes the bot and loads all the necessary data.
|
Initializes the bot and loads all the necessary data.
|
||||||
|
|
||||||
:param trigger: object of the trigger
|
|
||||||
:param config: (dictionary) config.toml as a dictionary of dictionaries
|
:param config: (dictionary) config.toml as a dictionary of dictionaries
|
||||||
:param logger: object of the logger
|
|
||||||
:param history_path: Path to the file with ID of the last retweeted
|
:param history_path: Path to the file with ID of the last retweeted
|
||||||
Tweet
|
Tweet
|
||||||
"""
|
"""
|
||||||
|
@ -46,7 +44,6 @@ class RetweetBot(object):
|
||||||
|
|
||||||
self.history_path = history_path
|
self.history_path = history_path
|
||||||
self.last_mention = self.get_history(self.history_path)
|
self.last_mention = self.get_history(self.history_path)
|
||||||
self.trigger = trigger
|
|
||||||
self.waitcounter = 0
|
self.waitcounter = 0
|
||||||
|
|
||||||
def get_api_keys(self):
|
def get_api_keys(self):
|
||||||
|
@ -85,7 +82,7 @@ class RetweetBot(object):
|
||||||
f.write(last_mention)
|
f.write(last_mention)
|
||||||
return int(last_mention)
|
return int(last_mention)
|
||||||
|
|
||||||
def save_last_mention(self):
|
def save_last(self):
|
||||||
""" Saves the last retweeted tweet in last_mention. """
|
""" Saves the last retweeted tweet in last_mention. """
|
||||||
with open(self.history_path, "w") as f:
|
with open(self.history_path, "w") as f:
|
||||||
f.write(str(self.last_mention))
|
f.write(str(self.last_mention))
|
||||||
|
@ -101,53 +98,53 @@ class RetweetBot(object):
|
||||||
self.waitcounter -= 1
|
self.waitcounter -= 1
|
||||||
return self.waitcounter
|
return self.waitcounter
|
||||||
|
|
||||||
def format_mastodon(self, status):
|
def crawl(self):
|
||||||
"""
|
|
||||||
Bridge your Retweets to mastodon.
|
|
||||||
|
|
||||||
:rtype: string
|
|
||||||
:param status: Object of a tweet.
|
|
||||||
:return: toot: text tooted on mastodon, e.g. "_b3yond: There are
|
|
||||||
uniformed controllers in the U2 at Opernhaus."
|
|
||||||
"""
|
|
||||||
toot = status.user.name + ": " + status.text
|
|
||||||
return toot
|
|
||||||
|
|
||||||
def crawl_mentions(self):
|
|
||||||
"""
|
"""
|
||||||
crawls all Tweets which mention the bot from the twitter rest API.
|
crawls all Tweets which mention the bot from the twitter rest API.
|
||||||
|
|
||||||
:return: list of Status objects
|
:return: reports: (list of report.Report objects)
|
||||||
"""
|
"""
|
||||||
|
reports = []
|
||||||
try:
|
try:
|
||||||
if not self.waiting():
|
if not self.waiting():
|
||||||
if self.last_mention == 0:
|
if self.last_mention == 0:
|
||||||
mentions = self.api.mentions_timeline()
|
mentions = self.api.mentions_timeline()
|
||||||
else:
|
else:
|
||||||
mentions = self.api.mentions_timeline(since_id=self.last_mention)
|
mentions = self.api.mentions_timeline(since_id=self.last_mention)
|
||||||
return mentions
|
for status in mentions:
|
||||||
|
text = re.sub("(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)", "", status.text)
|
||||||
|
reports.append(report.Report(status.author.screen_name,
|
||||||
|
"twitter",
|
||||||
|
text,
|
||||||
|
status.id,
|
||||||
|
status.created_at))
|
||||||
|
self.save_last()
|
||||||
|
return reports
|
||||||
except tweepy.RateLimitError:
|
except tweepy.RateLimitError:
|
||||||
logger.error("Twitter API Error: Rate Limit Exceeded", exc_info=True)
|
logger.error("Twitter API Error: Rate Limit Exceeded", exc_info=True)
|
||||||
self.waitcounter += 60*15 + 1
|
self.waitcounter += 60*15 + 1
|
||||||
except requests.exceptions.ConnectionError:
|
except requests.exceptions.ConnectionError:
|
||||||
logger.error("Twitter API Error: Bad Connection", exc_info=True)
|
logger.error("Twitter API Error: Bad Connection", exc_info=True)
|
||||||
self.waitcounter += 10
|
self.waitcounter += 10
|
||||||
|
except tweepy.TweepError:
|
||||||
|
logger.error("Twitter API Error: General Error", exc_info=True)
|
||||||
return []
|
return []
|
||||||
|
|
||||||
def retweet(self, status):
|
def repost(self, status):
|
||||||
"""
|
"""
|
||||||
Retweets a given tweet.
|
Retweets a given tweet.
|
||||||
|
|
||||||
:param status: A tweet object.
|
:param status: (report.Report object)
|
||||||
:return: toot: string of the tweet, to toot on mastodon.
|
:return: toot: string of the tweet, to toot on mastodon.
|
||||||
"""
|
"""
|
||||||
while 1:
|
while 1:
|
||||||
try:
|
try:
|
||||||
self.api.retweet(status.id)
|
self.api.retweet(status.id)
|
||||||
logger.info("Retweeted: " + self.format_mastodon(status))
|
logger.info("Retweeted: " + status.format())
|
||||||
if status.id > self.last_mention:
|
if status.id > self.last_mention:
|
||||||
self.last_mention = status.id
|
self.last_mention = status.id
|
||||||
return self.format_mastodon(status)
|
self.save_last()
|
||||||
|
return status.format()
|
||||||
except requests.exceptions.ConnectionError:
|
except requests.exceptions.ConnectionError:
|
||||||
logger.error("Twitter API Error: Bad Connection", exc_info=True)
|
logger.error("Twitter API Error: Bad Connection", exc_info=True)
|
||||||
sleep(10)
|
sleep(10)
|
||||||
|
@ -156,75 +153,81 @@ class RetweetBot(object):
|
||||||
logger.error("Twitter Error", exc_info=True)
|
logger.error("Twitter Error", exc_info=True)
|
||||||
if status.id > self.last_mention:
|
if status.id > self.last_mention:
|
||||||
self.last_mention = status.id
|
self.last_mention = status.id
|
||||||
|
self.save_last()
|
||||||
return None
|
return None
|
||||||
|
|
||||||
def tweet(self, post):
|
def post(self, status):
|
||||||
"""
|
"""
|
||||||
Tweet a post.
|
Tweet a post.
|
||||||
|
|
||||||
:param post: String with the text to tweet.
|
:param status: (report.Report object)
|
||||||
"""
|
"""
|
||||||
if len(post) > 280:
|
text = status.format()
|
||||||
post = post[:280 - 4] + u' ...'
|
if len(text) > 280:
|
||||||
|
text = status.text[:280 - 4] + u' ...'
|
||||||
while 1:
|
while 1:
|
||||||
try:
|
try:
|
||||||
self.api.update_status(status=post)
|
self.api.update_status(status=text)
|
||||||
return
|
return
|
||||||
except requests.exceptions.ConnectionError:
|
except requests.exceptions.ConnectionError:
|
||||||
logger.error("Twitter API Error: Bad Connection", exc_info=True)
|
logger.error("Twitter API Error: Bad Connection", exc_info=True)
|
||||||
sleep(10)
|
sleep(10)
|
||||||
|
|
||||||
def flow(self, to_tweet=()):
|
def flow(self, trigger, to_tweet=()):
|
||||||
""" The flow of crawling mentions and retweeting them.
|
""" The flow of crawling mentions and retweeting them.
|
||||||
|
|
||||||
:param to_tweet: list of strings to tweet
|
:param to_tweet: list of strings to tweet
|
||||||
:return list of retweeted tweets, to toot on mastodon
|
:return list of retweeted tweets, to toot on mastodon
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# Tweet the toots the Retootbot gives to us
|
# Tweet the reports from other sources
|
||||||
for post in to_tweet:
|
for post in to_tweet:
|
||||||
self.tweet(post)
|
self.post(post)
|
||||||
|
|
||||||
# Store all mentions in a list of Status Objects
|
# Store all mentions in a list of Status Objects
|
||||||
mentions = self.crawl_mentions()
|
mentions = self.crawl()
|
||||||
mastodon = []
|
|
||||||
|
# initialise list of strings for other bots
|
||||||
|
all_tweets = []
|
||||||
|
|
||||||
for status in mentions:
|
for status in mentions:
|
||||||
# Is the Text of the Tweet in the triggerlist?
|
# Is the Text of the Tweet in the triggerlist?
|
||||||
if self.trigger.is_ok(status.text):
|
if trigger.is_ok(status.text):
|
||||||
# Retweet status
|
# Retweet status
|
||||||
toot = self.retweet(status)
|
toot = self.repost(status)
|
||||||
if toot:
|
if toot:
|
||||||
mastodon.append(toot)
|
all_tweets.append(toot)
|
||||||
|
|
||||||
# save the id so it doesn't get crawled again
|
# Return Retweets for posting on other bots
|
||||||
if status.id > self.last_mention:
|
return all_tweets
|
||||||
self.last_mention = status.id
|
|
||||||
self.save_last_mention()
|
|
||||||
# Return Retweets for tooting on mastodon
|
|
||||||
return mastodon
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
# create an Api object
|
# get the config dict of dicts
|
||||||
with open('config.toml') as configfile:
|
with open('config.toml') as configfile:
|
||||||
config = toml.load(configfile)
|
config = toml.load(configfile)
|
||||||
|
|
||||||
|
# set log file
|
||||||
fh = logging.FileHandler(config['logging']['logpath'])
|
fh = logging.FileHandler(config['logging']['logpath'])
|
||||||
fh.setLevel(logging.DEBUG)
|
fh.setLevel(logging.DEBUG)
|
||||||
logger.addHandler(fh)
|
logger.addHandler(fh)
|
||||||
|
|
||||||
|
# initialise trigger
|
||||||
trigger = trigger.Trigger(config)
|
trigger = trigger.Trigger(config)
|
||||||
bot = RetweetBot(trigger, config)
|
|
||||||
|
# initialise twitter bot
|
||||||
|
bot = RetweetBot(config)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
while True:
|
while True:
|
||||||
bot.flow()
|
# :todo separate into small functions
|
||||||
|
bot.flow(trigger)
|
||||||
sleep(60)
|
sleep(60)
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
print("Good bye. Remember to restart the bot!")
|
print("Good bye. Remember to restart the bot!")
|
||||||
except:
|
except:
|
||||||
logger.error('Shutdown', exc_info=True)
|
logger.error('Shutdown', exc_info=True)
|
||||||
bot.save_last_mention()
|
bot.save_last()
|
||||||
try:
|
try:
|
||||||
mailer = sendmail.Mailer(config)
|
mailer = sendmail.Mailer(config)
|
||||||
mailer.send('', config['mail']['contact'],
|
mailer.send('', config['mail']['contact'],
|
||||||
|
|
|
@ -7,34 +7,51 @@ import sendmail
|
||||||
|
|
||||||
from retootbot import RetootBot
|
from retootbot import RetootBot
|
||||||
from retweetbot import RetweetBot
|
from retweetbot import RetweetBot
|
||||||
|
from mailbot import Mailbot
|
||||||
from trigger import Trigger
|
from trigger import Trigger
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
# read config in TOML format (https://github.com/toml-lang/toml#toml)
|
# read config in TOML format (https://github.com/toml-lang/toml#toml)
|
||||||
with open('config.toml') as configfile:
|
with open('config.toml') as configfile:
|
||||||
config = toml.load(configfile)
|
config = toml.load(configfile)
|
||||||
|
|
||||||
|
# set log file
|
||||||
logger = logging.getLogger()
|
logger = logging.getLogger()
|
||||||
fh = logging.FileHandler(config['logging']['logpath'])
|
fh = logging.FileHandler(config['logging']['logpath'])
|
||||||
fh.setLevel(logging.DEBUG)
|
fh.setLevel(logging.DEBUG)
|
||||||
logger.addHandler(fh)
|
logger.addHandler(fh)
|
||||||
|
|
||||||
trigger = Trigger(config)
|
trigger = Trigger(config)
|
||||||
mbot = RetootBot(config, trigger)
|
|
||||||
tbot = RetweetBot(trigger, config)
|
bots = []
|
||||||
|
|
||||||
|
if config["muser"]["enabled"] != "false":
|
||||||
|
bots.append(RetootBot(config))
|
||||||
|
if config["tuser"]["enabled"] != "false":
|
||||||
|
bots.append(RetweetBot(config))
|
||||||
|
if config["mail"]["enabled"] != "false":
|
||||||
|
bots.append(Mailbot(config))
|
||||||
|
|
||||||
try:
|
try:
|
||||||
statuses = []
|
statuses = []
|
||||||
while True:
|
while True:
|
||||||
statuses = mbot.retoot(statuses)
|
for bot in bots:
|
||||||
statuses = tbot.flow(statuses)
|
reports = bot.crawl()
|
||||||
time.sleep(60)
|
for status in reports:
|
||||||
|
if not trigger.is_ok(status.text):
|
||||||
|
continue
|
||||||
|
for bot2 in bots:
|
||||||
|
if bot == bot2:
|
||||||
|
bot2.repost(status)
|
||||||
|
else:
|
||||||
|
bot2.post(status)
|
||||||
|
time.sleep(60) # twitter rate limit >.<
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
print("Good bye. Remember to restart the bot!")
|
print("Good bye. Remember to restart the bot!")
|
||||||
except:
|
except:
|
||||||
logger.error('Shutdown', exc_info=True)
|
logger.error('Shutdown', exc_info=True)
|
||||||
tbot.save_last_mention()
|
for bot in bots:
|
||||||
|
bot.save_last()
|
||||||
try:
|
try:
|
||||||
mailer = sendmail.Mailer(config)
|
mailer = sendmail.Mailer(config)
|
||||||
mailer.send('', config['mail']['contact'],
|
mailer.send('', config['mail']['contact'],
|
||||||
|
|
Loading…
Reference in a new issue