don't log Mastodon 502 errors.

Building in CSRF prevention

README.md

@@ -152,19 +152,10 @@ echo "Enter your domain name into the following prompt:" && read DOMAIN
 # configure nginx
 sudo sed -r "s/example.org/$DOMAIN/g" deployment/example.org.conf > /etc/nginx/sites-enabled/$DOMAIN.conf
 
-# create folder for database
+# create folder for socket & database
 sudo mkdir /var/ticketfrei
 sudo chown www-data:www-data -R /var/ticketfrei
 
-# create folder for socket
-sudo mkdir /var/run/ticketfrei
-sudo chown -R www-data:www-data /var/run/ticketfrei
-sudo -s
-echo "mkdir /var/run/ticketfrei" >> /etc/rc.local
-echo "chown -R www-data:www-data /var/run/ticketfrei" >> /etc/rc.local
-echo "service ticketfrei-web restart" >> /etc/rc.local
-exit
-
 # change /etc/aliases permissions to be able to receive reports per mail
 sudo chown root:www-data /etc/aliases
 sudo chmod 664 /etc/aliases
@@ -187,17 +178,6 @@ sudo systemctl daemon-reload
 sudo systemctl start ticketfrei-backend.service
 ```
 
-### Backup
-
-For automated backups, you need to backup these files:
-
-* `/var/ticketfrei/db.sqlite`
-* `/srv/ticketfrei/config.toml`
-* `/etc/aliases`
-
-You can find an example how to do this with borgbackup in the deployment
-folder. Adjust it to your needs.
-
 ### Logs
 
 There are several logfiles which you can look at:
@@ -267,113 +247,3 @@ sudo chown $USER:$USER -R /var/log/ticketfrei
 ./frontend.py & ./backend.py &
 ```
 
-# Project History
-
-## Version 1
-
-- more of less hacked together during a mate-fueled weekend
-- backend-only, twitter & mastodon
-- just a script, which crawled & retweeted tweets, if they match a whitelist & blocklist
-- whitelist & blocklist were just 2 files
-
-## Version 2
-
-Reasons for the rewrite:
-- user management: Users should be able to run a Ticketfrei bot in their city
-  - without needing a server, without needing command line skills
-- more networks; not only Twitter & Mastodon, also Email & Telegram
-
-2 processes: backend & frontend.  
-The two Processes talk via a database.
-The two Processes have separate log files.
-Both processes take some config values from config.toml.
-
-### Backend
-
-The Backend takes care of crawling & spreading the reports.
-
-backend.py:
-- main loop which does the crawling & posting. 
-- loops through all cities in the database
-  - per city it tries all of the networks/bots:
-    - per network/bot it runs the crawl()-function to ask the social network for new reports
-    - then it checks whether the report is appropriate
-    - if yes, it posts the report via all networks/bots, which belong to the city.
-
-config.py: imports config values
-- Imports values from config.toml
-- If there is no config file it tries to use environment variables,
-- Apart from that it uses the default values.
-
-bot.py: bot parent Class
-- just the absolute minimum what a bot needs to be able to do: crawl + post
-- is never instantiated, only inherited from
-
-report.py: report Class
-- defines how reports are supposed to look like
-
-active_bots/mailbot.py as an example for how a network/bot works
-- crawl():
-  - mails arrive at an mbox file through exim4
-  - the bot checks whether they are new
-  - the bot generates a report object from the mail and returns it to the backend.py-loop
-- post():
-  - asks the database for the list of mails which want to receive reports for this city
-  - sends the report.text to those mail addresses
-
-
-### Frontend
-
-the architecture of the frontend is loosely oriented off [Model View
-Controller](https://blog.codinghorror.com/understanding-model-view-controller/).
-
-user.py (Model)
-- high-level interface to talk to the database
-- database calls; almost all values in the database are specific to a city/user
-- user.py is also a Class for frontend web authentication
-- user.py keeps the user-id, through which the frontend tracks authentication
-
-db.py (Model)
-- DB-Layout; creates the database if it doesn't exist yet.
-- holds some database calls which are not city-specific.
-
-frontend.py (Controller): bottle web application
-- handles POST/GET requests
-- talks to the database through user.py
-- everyone can look at the pages, and register
-- but only authenticated users can login and change settings
-
-session.py: User Authentication
-- takes care of session cookies and "403 unauthenticated" error messages
-
-sendmail.py: helper script to send mails
-- sends all mails the frontend, backend, and bots need to send
-
-static/
-- css, images, javascript for the login form etc.
-
-template/ (view)
-- base for the HTML generation, uses the bottle-template-framework
-- wrapper.tpl is the base template for every other template
-
-
-### active_bots: how to implement a new network
-
-If you want to write a new bot, e.g. a Wire-Bot, you have to take these steps:
-
-- look for a python-library which can talk to Wire
-- the city/users have to provide authentication details; this needs a form in
-  the settings
-  - depending on the network either a password, a token, or an implementation
-    of the OAuth login flow
-- the backend needs to crawl messages from the network, & post reports to the
-  network
-
-Files you need to change:
-
-1. active_bots/wire.py - crawl & post functions
-2. settings.tpl - form to authenticate to the network & possible network specific settings.
-3. frontend.py - routes for the forms you added to settings.tpl
-4. db.py - database layout, to store the account credentials/tokens, and to save which message you have last seen
-5. user.py - database calls to get or set values
-

active_bots/mailbot.py

@@ -10,7 +10,7 @@ from bot import Bot
 from config import config
 from db import db
 
-logger = logging.getLogger("main")
+logger = logging.getLogger(__name__)
 
 
 class Mailbot(Bot):
@@ -58,16 +58,30 @@ def make_report(msg, user):
     date = get_date_from_header(msg['Date'])
 
     author = msg['From']  # get mail author from email header
-    for part in msg.walk():
-        if part.get_content_type() == 'text/plain':
-            text = part.get_payload()
-        elif part.get_content_type() == 'text/html':
-            text = re.sub(r'<[^>]*>', '', msg.get_payload())
-    try:
+
+    if msg.is_multipart():
+        text = []
+        for part in msg.get_payload():
+            if part.get_content_type() == "text":
+                text.append(part.get_payload())
+            elif part.get_content_type() == "application/pgp-signature":
+                pass  # ignore PGP signatures
+            elif part.get_content_type() == "multipart/mixed":
+                for p in part:
+                    if isinstance(p, str):
+                        text.append(p)
+                    elif p.get_content_type() == "text":
+                        text.append(part.get_payload())
+                    else:
+                        logger.error("unknown MIMEtype: " +
+                                     p.get_content_type())
+            else:
+                logger.error("unknown MIMEtype: " +
+                             part.get_content_type())
+        text = '\n'.join(text)
+    else:
+        text = msg.get_payload()
     post = report.Report(author, "mail", text, None, date)
-    except UnboundLocalError:
-        logger.error('No suitable message body')
-        return
     user.save_seen_mail(date)
     return post
 

active_bots/mastodonbot.py

@@ -2,12 +2,12 @@
 
 from bot import Bot
 import logging
-import mastodon
+from mastodon import Mastodon, MastodonAPIError
 import re
 from report import Report
 
 
-logger = logging.getLogger("main")
+logger = logging.getLogger(__name__)
 
 
 class MastodonBot(Bot):
@@ -19,65 +19,22 @@ class MastodonBot(Bot):
         """
         mentions = []
         try:
-            m = mastodon.Mastodon(*user.get_masto_credentials())
+            m = Mastodon(*user.get_masto_credentials())
         except TypeError:
-            # No Mastodon Credentials in database.
+            # logger.error("No Mastodon Credentials in database.", exc_info=True)
             return mentions
         try:
             notifications = m.notifications()
-        except mastodon.MastodonNetworkError:
-            logger.error("Mastodon Network Error.")
+        except MastodonAPIError:
+            # Not tested yet, as we don't have a dysfunctional Mastodon instance.
+            for arg in MastodonAPIError.args:
+                if arg == 502:
+                    # Don't log 502 errors
                     return mentions
-        except mastodon.MastodonAPIError:
-            try:
-                logger.error("Mastodon API Error: " + m.instance()['urls']['streaming_api'] + ", city: " + str(user.uid))
-            except mastodon.MastodonServerError:
-                logger.error("Mastodon Server Error 500, can't get instance.")
-            except mastodon.MastodonVersionError:
-                logger.error("Mastodon Server Error 500, server version too low.")
-            return mentions
-        except mastodon.MastodonInternalServerError:
-            try:
-                logger.error("Mastodon Error: 500. Server: " + m.instance()['urls']['streaming_api'])
-            except mastodon.MastodonServerError:
-                logger.error("Mastodon Server Error 500, can't get instance.")
-            except mastodon.MastodonVersionError:
-                logger.error("Mastodon Server Error 500, server version too low.")
-            return mentions
-        except mastodon.MastodonBadGatewayError:
-            try:
-                logger.error("Mastodon Error: 502. Server: " + m.instance()['urls']['streaming_api'])
-            except mastodon.MastodonServerError:
-                logger.error("Mastodon Server Error 502, can't get instance.")
-            except mastodon.MastodonVersionError:
-                logger.error("Mastodon Server Error 502, server version too low.")
-            return mentions
-        except mastodon.MastodonServiceUnavailableError:
-            try:
-                logger.error("Mastodon Error: 503. Server: " + m.instance()['urls']['streaming_api'])
-            except mastodon.MastodonServerError:
-                logger.error("Mastodon Server Error 503, can't get instance.")
-            except mastodon.MastodonVersionError:
-                logger.error("Mastodon Server Error 503, server version too low.")
-            return mentions
-        except mastodon.MastodonGatewayTimeoutError:
-            try:
-                logger.error("Mastodon Error: 504. Server: " + m.instance()['urls']['streaming_api'])
-            except mastodon.MastodonServerError:
-                logger.error("Mastodon Server Error 504, can't get instance.")
-            except mastodon.MastodonVersionError:
-                logger.error("Mastodon Server Error 504, server version too low.")
-            return mentions
-        except mastodon.MastodonServerError:
-            try:
-                logger.error("Unknown Mastodon Server Error. Server: " + m.instance()['urls']['streaming_api'], exc_info=True)
-            except mastodon.MastodonServerError:
-                logger.error("Unknown Mastodon Server Error.", exc_info=True)
-            except mastodon.MastodonVersionError:
-                logger.error("Unknown Mastodon Server Error.", exc_info=True)
+
+            logger.error("Unknown Mastodon API Error.", exc_info=True)
             return mentions
         for status in notifications:
-            try:
             if (status['type'] == 'mention' and
                     not user.toot_is_seen(status['status']['uri'])):
                 # save state
@@ -99,13 +56,11 @@ class MastodonBot(Bot):
                                            text,
                                            status['status']['id'],
                                            status['status']['created_at']))
-            except TypeError:
-                pass
         return mentions
 
     def post(self, user, report):
         try:
-            m = mastodon.Mastodon(*user.get_masto_credentials())
+            m = Mastodon(*user.get_masto_credentials())
         except TypeError:
             return  # no mastodon account for this user.
         if report.source == self:

active_bots/telegrambot.py

@@ -3,7 +3,8 @@ import logging
 from report import Report
 from twx.botapi import TelegramBot as Telegram
 
-logger = logging.getLogger("main")
+
+logger = logging.getLogger(__name__)
 
 
 class TelegramBot(Bot):
@@ -12,10 +13,9 @@ class TelegramBot(Bot):
         seen_tg = user.get_seen_tg()
         try:
             updates = tb.get_updates(offset=seen_tg + 1,
-                                     allowed_updates="message",
-                                     timeout=5).wait()
+                                     allowed_updates="message").wait()
         except TypeError:
-            updates = tb.get_updates(timeout=5).wait()
+            updates = tb.get_updates().wait()
         reports = []
         if updates == None:
             return reports
@@ -23,7 +23,6 @@ class TelegramBot(Bot):
             # return when telegram returns an error code
             if update in [303, 404, 420, 500, 502]:
                 return reports
-            # log unusual telegram error messages
             if isinstance(update, int):
                 try:
                     logger.error("City " + str(user.uid) +
@@ -32,25 +31,7 @@ class TelegramBot(Bot):
                 except TypeError:
                     logger.error("Unknown Telegram error code: " + str(update))
                 return reports
-            # save the last message, so it doesn't get crawled again
             user.save_seen_tg(update.update_id)
-            # skip if message is None
-            if update.message is None:
-                continue
-            # complain if message is a photo
-            if update.message.photo is not None:
-                tb.send_message(
-                    update.message.sender.id,
-                    "Sending Photos is not supported for privacy reasons. Can "
-                    "you describe it as text instead?")
-                continue
-            # complain if message is a media file
-            if update.message.text is None:
-                tb.send_message(
-                    update.message.sender.id,
-                    "We only support text reporting for privacy reasons. Can "
-                    "you describe it as text instead?")
-                continue
             if update.message.text.lower() == "/start":
                 user.add_telegram_subscribers(update.message.sender.id)
                 tb.send_message(

active_bots/twitterDMs.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+
+import logging
+import tweepy
+import re
+import requests
+import report
+from time import time
+from bot import Bot
+
+
+logger = logging.getLogger(__name__)
+
+
+class TwitterDMListener(Bot):
+
+    def get_api(self, user):
+        keys = user.get_twitter_credentials()
+        auth = tweepy.OAuthHandler(consumer_key=keys[0],
+                                   consumer_secret=keys[1])
+        auth.set_access_token(keys[2],  # access_token_key
+                              keys[3])  # access_token_secret
+        return tweepy.API(auth, wait_on_rate_limit=True)
+
+    def crawl(self, user):
+        """
+        crawls all Tweets which mention the bot from the twitter rest API.
+
+        :return: reports: (list of report.Report objects)
+        """
+        reports = []
+        try:
+            if user.get_last_twitter_request() + 60 > time():
+                return reports
+        except TypeError:
+            user.set_last_twitter_request(time())
+        try:
+            api = self.get_api(user)
+        except TypeError:
+            return reports  # no twitter account for this user.
+        last_dm = user.get_seen_dm()
+        try:
+            if last_dm is None:
+                mentions = api.direct_messages()
+            else:
+                mentions = api.direct_messages(since_id=last_dm[0])
+            user.set_last_twitter_request(time())
+            for status in mentions:
+                text = re.sub(
+                       "(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9-_]+)",
+                       "", status.text)
+                reports.append(report.Report(status.author.screen_name,
+                                             "twitterDM",
+                                             text,
+                                             status.id,
+                                             status.created_at))
+            user.save_seen_dm(last_dm)
+            return reports
+        except tweepy.RateLimitError:
+            logger.error("Twitter API Error: Rate Limit Exceeded",
+                         exc_info=True)
+            # :todo implement rate limiting
+        except requests.exceptions.ConnectionError:
+            logger.error("Twitter API Error: Bad Connection", exc_info=True)
+        except tweepy.TweepError as terror:
+            # Waiting for https://github.com/tweepy/tweepy/pull/1109 to get
+            # merged, so direct messages work again
+            if terror.api_code == 34:
+                return reports
+            logger.error("Twitter API Error: General Error", exc_info=True)
+        return reports
+
+    def post(self, user, report):
+        pass

active_bots/twitterbot.py

@@ -9,7 +9,7 @@ import report
 from bot import Bot
 
 
-logger = logging.getLogger("main")
+logger = logging.getLogger(__name__)
 
 
 class TwitterBot(Bot):
@@ -67,13 +67,13 @@ class TwitterBot(Bot):
         except requests.exceptions.ConnectionError:
             logger.error("Twitter API Error: Bad Connection", exc_info=True)
         except tweepy.TweepError:
-            logger.error("Twitter API Error: General Error. User: " + str(user.uid), exc_info=True)
+            logger.error("Twitter API Error: General Error", exc_info=True)
         return []
 
     def post(self, user, report):
         try:
             api = self.get_api(user)
-        except TypeError:
+        except IndexError:
             return  # no twitter account for this user.
         try:
             if report.source == self:

backend.py

@@ -5,7 +5,7 @@ from config import config
 from db import db
 import logging
 from sendmail import sendmail
-from time import sleep
+
 
 def shutdown():
     try:
@@ -16,14 +16,11 @@ def shutdown():
 
 
 if __name__ == '__main__':
-    logger = logging.getLogger("main")
-    logger.setLevel(logging.DEBUG)
+    logger = logging.getLogger()
     fh = logging.FileHandler('/var/log/ticketfrei/backend.log')
-    formatter = logging.Formatter('%(asctime)s %(levelname)8s: %(message)s')
-    fh.setFormatter(formatter)
+    fh.setLevel(logging.DEBUG)
     logger.addHandler(fh)
 
-    logger.info("Backend Daemon was started...")
 
     bots = []
     for ActiveBot in active_bots.__dict__.values():
@@ -34,14 +31,12 @@ if __name__ == '__main__':
         while True:
             for user in db.active_users:
                 for bot in bots:
-                    sleep(1)
                     reports = bot.crawl(user)
                     for status in reports:
                         if not user.is_appropriate(status):
                             logger.info("Inaproppriate message: %d %s %s" % (user.uid, status.author, status.text))
                             continue
                         for bot2 in bots:
-                            sleep(1)
                             bot2.post(user, status)
                             logger.info("Resent: %d %s %s" % (user.uid, status.author, status.text))
     except Exception:

db.py

@@ -1,13 +1,12 @@
 from config import config
 import jwt
 import logging
-from os import urandom, system
+from os import urandom
 from pylibscrypt import scrypt_mcf
 import sqlite3
-from time import sleep, time
 
 
-logger = logging.getLogger("main")
+logger = logging.getLogger(__name__)
 
 
 class DB(object):
@@ -20,19 +19,6 @@ class DB(object):
         return self.cur.execute(*args, **kwargs)
 
     def commit(self):
-        start_time = time()
-        while 1:
-            try:
-                self.conn.commit()
-                break
-            except sqlite3.OperationalError as error:
-                # another thread may be writing, give it a chance to finish
-                sleep(0.5)
-                logger.exception()
-                if time() - start_time > 5:
-                    # if it takes this long, something is wrong
-                    system("rcctl restart frontend_daemon")
-                    logger.warning("frontend_daemon is getting restarted")
         self.conn.commit()
 
     def close(self):
@@ -259,10 +245,6 @@ u\d\d?"""
             uid = json['uid']
         with open("/etc/aliases", "a+") as f:
             f.write(city + ": " + config["mail"]["mbox_user"] + "\n")
-            try:
-                os.system("newaliases")
-            except:
-                logger.exception()
         self.execute("INSERT INTO email (user_id, email) VALUES(?, ?);",
                      (uid, json['email']))
         self.execute("""INSERT INTO telegram_accounts (user_id, apikey,

deployment/backend_daemon

@@ -1,15 +0,0 @@
-#!/bin/ksh
-
-daemon="/usr/local/bin/python3 /srv/ticketfrei/backend.py"
-daemon_user="root"
-
-. /etc/rc.d/rc.subr
-
-rc_bg=YES
-rc_reload=NO
-
-rc_start() {
-        rc_exec "cd /srv/ticketfrei; /usr/bin/nice -n15 ${daemon} ${daemon_flags}"
-}
-
-rc_cmd $1

deployment/borgbackup.sh

@@ -1,15 +0,0 @@
-#!/bin/ksh
-. /etc/borg-env
-export BORG_REPO=nathan@nephilim:repositories-borg/ticketfrei
-export BORG_RSH="ssh \
--o TCPKeepAlive=no \
--o ServerAliveInterval=15 \
--o ServerAliveCountMax=10 \
--o Compression=no"
-
-rcctl stop backend_daemon
-rcctl stop frontend_daemon
-/usr/local/bin/borg create --stats ::'backup{now:%Y%m%d-%H%M}' /srv/ticketfrei /var/ticketfrei /etc
-rcctl start backend_daemon
-rcctl start frontend_daemon
-

deployment/example.org.conf

@@ -0,0 +1,31 @@
+server {
+
+    listen 443 ssl;
+    server_name example.org;
+    ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem;
+    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
+    ssl_ciphers         HIGH:!aNULL:!MD5;
+
+    access_log /var/log/nginx/example.org_access.log;
+    error_log /var/log/nginx/example.org_error.log;
+
+    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
+
+    location / {
+        include uwsgi_params;
+
+        uwsgi_pass unix:///var/run/ticketfrei/ticketfrei.sock;
+    }
+
+    location /.well-known/acme-challenge {
+        root /var/www/acme;
+    }
+}
+server {
+    listen 80;
+    listen [::]:80;
+    server_name example.org;
+    return 301 https://$server_name$request_uri;
+}
+

deployment/frontend_daemon

@@ -1,15 +0,0 @@
-#!/bin/ksh
-
-daemon="/usr/local/bin/python3 /srv/ticketfrei/frontend.py"
-daemon_user="frontend"
-
-. /etc/rc.d/rc.subr
-
-rc_bg=YES
-rc_reload=NO
-
-rc_start() {
-        rc_exec "env > /tmp/envars; cd /srv/ticketfrei; ${daemon} ${daemon_flags}"
-}
-
-rc_cmd $1

deployment/ticketfrei-backend.service

@@ -0,0 +1,17 @@
+[Unit]
+Description=Ticketfrei Backend
+After=syslog.target network.target
+
+[Service]
+WorkingDirectory=/srv/ticketfrei
+ExecStart=/srv/ticketfrei/bin/python3 backend.py
+# Requires systemd version 211 or newer
+#RuntimeDirectory=uwsgi
+Restart=always
+KillSignal=SIGQUIT
+Type=simple
+StandardError=syslog
+NotifyAccess=all
+
+[Install]
+WantedBy=multi-user.target

deployment/ticketfrei-web.service

@@ -0,0 +1,17 @@
+[Unit]
+Description=Ticketfrei Web Application
+After=syslog.target network.target
+
+[Service]
+WorkingDirectory=/srv/ticketfrei
+ExecStart=/usr/bin/uwsgi --ini /srv/ticketfrei/deployment/uwsgi.ini
+# Requires systemd version 211 or newer
+RuntimeDirectory=uwsgi
+Restart=always
+KillSignal=SIGQUIT
+Type=notify
+StandardError=syslog
+NotifyAccess=all
+
+[Install]
+WantedBy=multi-user.target

requirements.txt

@@ -1,9 +0,0 @@
-tweepy
-pytoml
-Mastodon.py
-bottle
-pyjwt
-pylibscrypt
-Markdown
-twx
-gitpython

sendmail.py

@@ -8,7 +8,7 @@ import smtplib
 from socket import getfqdn
 
 
-logger = logging.getLogger("main")
+logger = logging.getLogger(__name__)
 
 
 def sendmail(to, subject, city=None, body=''):

user.py

@@ -5,9 +5,7 @@ import jwt
 from mastodon import Mastodon
 from pylibscrypt import scrypt_mcf, scrypt_mcf_check
 from os import urandom
-import logging
 
-logger = logging.getLogger("main")
 
 class User(object):
     def __init__(self, uid):
@@ -76,7 +74,6 @@ class User(object):
                 break
         else:
             # no pattern matched
-            logger.error("Message didn't trigger goodlist: " + report.text)
             return False
         default_badwords = """
 bitch
@@ -96,15 +93,12 @@ schlitz
         db.execute("SELECT words FROM badwords WHERE user_id=?;",
                    (self.uid, ))
         badwords = db.cur.fetchone()
-        for word in report.text.lower().split():
-            if word in badwords[0].splitlines():
-                logger.error("Word " + word + " triggered the spam filter on message: " + report.text)
+        for word in report.text.lower().splitlines():
+            if word in badwords:
                 return False
-        for word in report.text.lower().split():
-            if word in default_badwords.splitlines():
-                logger.error("Word " + word + " triggered the spam filter on message: " + report.text)
+        for word in default_badwords.splitlines():
+            if word in badwords:
                 return False
-        logger.info("Valid report: " + report.text + " | username: " + report.author)
         return True
 
     def get_last_twitter_request(self):
@@ -287,8 +281,6 @@ schlitz
                            user_id, client_id, client_secret
                            ) VALUES(?, ?, ?);""",
                    (self.uid, access_token, access_token_secret))
-        db.execute("""INSERT INTO seen_tweets(user_id, tweet_id) VALUES (?, ?);""",
-                   (self.uid, 0))
         db.commit()
 
     def get_twitter_token(self):