[mediawiki-setup] / Mediawiki_Setup_Guide

== Introduction ==

'''Currently undergoing revisions to upgrade from mediawiki version 1.23 to 1.27.
Should be done on 5/30/2016'''

'''tldr''': Automated setup on gnu/linux after you set options in [[#Dependent Variables and Functions]], except email. Any code which is not ready to be run has bold text saying so immediately before it.

''' Goals / Why use this guide? '''

* Good recommendations. Official docs mostly avoid recommendations among a myriad of possibilities
* Closely references & supplements official documentation
* Explicit automation support wherever practical
* Used to setup this site (style optional)
* Contributions welcome. This site will be updated/tested with changes to this document!
* Support for multiple linux distros
* Holistic scope (backups, server setup), but sections stand on their own
* Explicit support for production & local testing instance. Additions for production like https and web analytics.
* Edits to this page are closely monitored by the original author.

'''Assumptions'''

* Self hosting, single Linux system with Bash shell
* Root shell is assumed throughout
* Code blocks are [https://en.wikipedia.org/wiki/Idempotent idempotent] unless stated otherwise

'''Version Support'''

Very minor adjustments needed for other distros. Help expand this list.
* Mediawiki 1.23
* Fedora 20 (tested)
* Ubuntu 14.04 (tested)
* Debian 7

== Production Server Prerequisites ==

'''Getting a Server & a Domain'''

The most common route and the one taken by this site is buying a domain name from a site like namecheap, and a cheap vps from companies like linode or digital ocean. They have good getting started guides which mostly apply beyond their own sites.

'''Email Setup'''

Setting up email can be an involved process. Mediawiki is perfectly happy to disable email with 1 setting (no password reminders or notifications), but it is a nice feature to have. You could run your own mail server (on the mediawiki server, or elsewhere), or use one of many services which sends mail for very cheap, or free within limits (popular examples [http://www.mailgun.com/ mailgun], [https://mandrillapp.com/ mandrill], [http://www.mailjet.com/pricing mailjet], [https://aws.amazon.com/ses/ aws]), or connect  to a full featured send/receive mail provider like [https://fastmail.com fastmail] (this server is setup that way). How we did that is for a future wiki page.

If you are not setting up your server to send mail with a program that uses the default sendmail interface, see these pages when you are configuring mediawiki: [[mediawikiwiki:Manual:$wgEnableEmail|Manual:$wgEnableEmail]], [https://www.mediawiki.org/wiki/Configuration_settings#Email_settings Manual:Email_settings], [[mediawikiwiki:Manual:$wgSMTP|Manual:$wgSMTP]]

== Stopping Spam ==

There is a balance between effective anti-spam measures and blocking/annoying contributors. Mediawiki documentation on how to combat spam, is not very good, but it has improved over time: [https://www.mediawiki.org/wiki/Manual:Combating_spam manual: Combating Spam]. It's possible for a spammer to quickly make thousands of edits, and there is no good documentation on purging lots of spam, so you should have a good strategy up front. My current strategy is 3 fold, and is limited to small/medium wiki's:

* Find new spam quickly, revert it & ban the user.
** Watch, and get notified of changes on all primary content pages: Special:Preferences, Bottom of the page, set an email address, then turn on "Email me also for minor edits of pages and files."
** Use a rss/atom feed reader, and subscribe to recent changes across the wiki. Newer browsers have an rss feed subscribe button, you can click after going to Special:RecentChanges. If that is not available, you can construct the proper url based on [https://meta.wikimedia.org/wiki/Help:Recent_changes#Web_feed these instructions].
* Require registration to edit, and a custom captcha question on registration.
* Install all non-user inhibiting anti-spam extensions / settings that take a reasonable amount of time to figure out.

== Download all ready to use source code blocks into a single script ==

Source block that are not ready to use have a bold warning just before them (and a tag on the block) and are skipped.
''' Requires customization: outputs to stdout'''
<source lang="bash" type="example">
start=' *<source lang="bash"> *'
end=' *<\/source> *'
ruby <<'EOF' | sed -rn "/^$start$/,/^$end$/{s/^$start|$end$/# \0/;p}"
require 'json'
puts JSON.parse(`curl 'https://ofswiki.org/w/api.php?\
action=query&titles=Mediawiki_Setup_Guide&prop=revisions&rvprop=content&\
format=json'`.chomp)['query']['pages'].values[0]['revisions'][0]['*']
EOF
</source>

== Dependent Variables and Functions ==

# Set variables below
# Save the code in this section to a file (~/mw_vars is suggested)
# Source it at the beginning of scripts containing later commands
# Source it from your .bashrc file while you are setting up Mediawiki

'''Requires customization:'''
<source lang="bash" type="example">
# Replease REPLACE_ME as appropriate

export mwdescription="Opinionated Free Software Wiki"

# username/pass of the first wiki admin user
export wikiuser="Ian Kelling"
export wikipass=REPLACE_ME

# root password for the mysql database
export dbpass=REPLACE_ME

# password for piwik web analytics admin
export piwik_pass=REPLACE_ME

# git branch for mediawiki + extensions
# branch names: https://git.wikimedia.org/branches/mediawiki%2Fcore.git
export mw_branch=REL1_27

# customize these questions to something your contributors would know,
# is not super common / scriptable, and doesn't have the answer in the question
captchaArray() {
    if ! grep -Fx '$localSettingsQuestyQuestions = array (' $mwc
    tee -a $mwc <<'EOF'
$localSettingsQuestyQuestions = array (
    "What is the name of the wiki software this site (and wikipedia) uses?" => "Mediawiki",
    "What does f in ofswiki.org stand for?" => "Free"
);
EOF
}

# As set by gui installer when choosing cc by sa.
export mw_RightsUrl="https://creativecommons.org/licenses/by-sa/4.0/";
export mw_RightsText="Creative Commons Attribution-ShareAlike";
export mw_RightsIcon="$wgResourceBasePath/resources/assets/licenses/cc-by-sa.png";


# domain name and install paths. These are examples, change as needed.
export mwfamily=ofswiki # short alphanumeric name for pywikibot
export mwdomain=ofswiki.org
export mw=/var/www/$mwdomain/html/w


# wiki sender address / wiki & wiki server contact email.
# see email section for more info on email
export mw_email="admin@$mwdomain"
</source>

Here we define some small useful bash functions. This should be part of the same file.

<source lang="bash">
# for convenience, Mediawiki config file
mwc="$mw/LocalSettings.php"

if isdeb; then
    apache_user=www-data
else
    apache_user=apache
fi

# get and reset an extension/skin repository, and enable it
mw-ext () { mw-extra extension $@; }
mw-skin() { mw-extra skin $@; }

mw-extra() {
    local type=${1}s # extension or skin
    shift
    local clone=true
    if [[ $1 == -g ]]; then
        clone=false
        shift
    fi
    local ext
    for ext in "$@"; do
        if $clone; then
            local original_pwd="$PWD"
            # it's ok that this fails if we already have it
            url=https://git.wikimedia.org/git/mediawiki
            target=$mw/$type/$ext
            git clone $url/$type/$ext.git $target
            if ! cd $target; then
                echo "mw-ext error: failed cd $mw/extensions/$ext";
                exit 1
            fi
            git fetch
            git checkout -qf origin/$mw_branch || git checkout -qf origin/master
            git clean -xffd
            cd "$original_pwd"
        fi
        case $type in
            extensions)
                if [[ -e $target/extension.json ]]; then
                    # new style extension. remove old style declaration
                    sed -i '#^require_once( "\\\$IP/extensions/\$ext/\$ext\.php" );#d' $mwc
                    teeu $mwc <<EOF
wfLoadExtension( '$ext' );
EOF
                else
                    teeu $mwc <<EOF
require_once( "\$IP/extensions/$ext/$ext.php" );
EOF
                fi
                ;;
            skins)
                sed -i '/^wfLoadSkin/d' $mwc
                sed -i '/^\$wgDefaultSkin/d' $mwc
                teeu $mwc <<EOF
$wgDefaultSkin = "${ext,,*}";
wfLoadSkin( 'Vector' );
EOF
                ;;
        esac
    done
    # --quick is quicker than default flags,
    # but still add a sleep to make sure everything works right
    sudo -u $apache_user php $mw/maintenance/update.php -q --quick; sleep 1
}
</source>

== Install Mediawiki Dependencies ==

The best way to get core dependencies is to install the mediawiki package itself. Nothing about it will get in the way of using a version from upstream.

[[mediawikiwiki:Main Page|Mediawiki Main Page]]: the beginning of the official docs.

[[mediawikiwiki:Manual:Installation_requirements|Manual:Installation_requirements]]: Overview of installation requirements.



<source lang="bash">
if isdeb; then
    apt-get update
    # noninteractive to avoid mysql password prompt
    DEBIAN_FRONTEND=noninteractive apt-get -y install mediawiki ImageMagick php5-mysqlnd php-apc
    service apache2 restart
else
    # fedora deps are missing a database, so some is translated from debian packages
    yum -y install mediawiki ImageMagick php-mysqlnd php-pecl-apcu mariadb-server

    systemctl restart mariadb.service
    systemctl enable mariadb.service
    systemctl enable httpd.service
    systemctl restart httpd.service
fi


# slightly different depending on if we already set the root pass
if echo exit|mysql -u root -p"$dbpass"; then
    # answer interactive prompts:
    # mysql root pass, change pass? no, remove anon users? (default, yes)
    # disallow remote root (default, yes), reload? (default, yes)
    echo -e "$dbpass\nn\n\n\n\n" | mysql_secure_installation
else
    # I had 1 less newline at the start when doing ubuntu 14.04,
    # compared to debian 8, so can't say this is especially portable.
    # It won't hurt if it fails.
    echo -e "\n\n$dbpass\n$dbpass\n\n\n\n\n" | mysql_secure_installation
fi
</source>


'''Skippable notes'''


php[5]-mysqlnd is a faster mysql driver package, but the default in debian php-mysql, appparently because some non-mediawiki packages are not compatible with it. If you run into this issue, simply use the php-mysql package.


Additional packages rational
* ImageMagick is [https://www.mediawiki.org/wiki/Manual:Image_administration#Image_thumbnailing recommended].
* Gui install and [[mediawikiwiki:Manual:Cache]] recomend the apc package.
* Clamav for virus scanning of uploads is mentioned in the mediawiki manual. However, wikipedia doesn't seem to do it, so it doesn't seem like it's worth bothering. It also makes uploading a set of images take twice as long on broadband.

== Install Mediawiki ==


Here, we [[mediawikiwiki:Download_from_Git]], or reset our installation if it is already there, and create the wiki database. [[mediawikiwiki:Manual:Installing_MediaWiki]]

<source lang="bash">
mkdir -p $mw
cd $mw
# this will just fail if it already exists which is fine
git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git .
# to see available branches: https://www.mediawiki.org/wiki/Version_lifecycle
# and
# git branch -r
git checkout -f origin/$mw_branch
git clean -ffxd
# Get the php libraries wmf uses. Based on:
# https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries
git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git
cd vendor
git checkout -f origin/$mw_branch
cd ..

# Drop any previous database which may have been installed while testing.
# If upgrading, we should have a db backup which will get restored.
# https://www.mediawiki.org/wiki/Manual:Upgrading
mysql -u root -p$dbpass <<'EOF'
drop database my_wiki;
exit
EOF
php $mw/maintenance/install.php --pass $wikipass --scriptpath /w \
    --dbuser root --dbpass $dbpass "$mwdescription" "$wikiuser"
teeu $mwc <<'EOF'
# lock down the wiki to only the initial owner until anti-spam measures are put in place
# limit edits to registered users
$wgGroupPermissions['*']['edit'] = false;
# don't allow any account creation
$wgGroupPermissions['*']['createaccount'] = false;
EOF
</source>


Note: When testing, you may need to clear the apc cache to see changes take effect in the browser. Simplest solution is
just restart apache. http://stackoverflow.com/questions/911158/how-to-clear-apc-cache-entries

''' Skippable Notes'''

If we wanted to reset our installation, but leave the extension repositories alone, alter the command above to be <code>git clean -fxd</code>

'''Rational for choosing git sources'''

Upstream vs distro packages. Upstream is responsive, and it's distributed within a single directory, so packaging does not integrate with the distro's filesystem. The only potential value would be less bugs by using stable versions, but we choose not to make that tradeoff.

Why use git over zip file releases? Mediawiki supports git usage through release branches which get post-release fixes. This means we can auto-update, get more granular fixes, easier to manage updates, and rollbacks.

== Get Piwik Files ==

Because it's really fun to look at your traffic stats. This is only needed for a production site. We get these files now so that we can take them into account when configuring apache in the next section.

[http://piwik.org/docs/installation-maintenance/ Overall installation guide]

<source lang="bash">
tmpdir="$(mktemp -d)"
cd $tmpdir
wget http://builds.piwik.org/piwik.zip
if isdeb; then apt-get -y install unzip; else yum -y install unzip; fi
unzip -q piwik.zip
# gui installer suggested command
if isdeb; then
    chown -R www-data:www-data piwik
else
    chown -R apache:apache piwik
fi
# remove any existing directory
rm -rf $mw/../analytics
mv piwik $mw/../analytics
rm -rf $tmpdir

</source>

== Configure Apache ==

Note, non-debian based installs: modify instructions below to use /etc/httpd/conf.d/$mwdomain.conf, and don't run a2ensite.

I use scripts I maintains separately to setup Let's Encrypt certificates and apache config: (url pending).

If you are doing a test setup on your local machine, you can make your domain resolve to your local test installation, then remove it later when you are done. Note, you will need non-local site to get Let's Encrypt certificates, and then transfer them locally, or disable ssl from the apache config (neither is covered here) and replace all instances of https in these instructions with http. Another option is to get a cheap 2 dollar domain for your test site.

'''Not for production:'''
<source lang="bash" type="example">
teeu /etc/hosts<<<"127.0.0.1 $mwdomain"
</source>

To not use my scripts, and still use Let's Encrypt: follow this doc page: https://letsencrypt.org/getting-started/. It's a little long winded, so I would boil it down to this:

'''Optional & requires additional steps:'''
<source lang="bash" type="example">
git clone https://github.com/certbot/certbot
cd certbot
./certbot-auto apache
cd /etc/apache/sites-available
mv 000-default-le-ssl.conf $mwdomain.conf
rm ../sites-enabled/000-default-le-ssl.conf
# edit $mwdomain.conf, so documentroot is /var/www/$mwdomain/html
# and ServerName is $mwdomain
a2ensite $mwdomain.conf
</source>
Then, copy the input to apache-site below and insert it into the apache config.

Here, I use my scripts to setup apache. It's behind nginx because the same server hosts other sites which are not convenient to use the same apache instance.
<source lang="bash">
l=$mw/../../logs
mkdir $l

acme-tiny-wrapper $mwdomain
dd of=/etc/apache2/ports.conf <<'EOF'
Listen 8082
EOF
{ cat <<EOF
  ServerAdmin $mw_email
  RewriteEngine On
  # make the site's root url go to our main page
  RewriteRule ^/?wiki(/.*)?\$ %{DOCUMENT_ROOT}/w/index.php [L]
  # use short urls https://www.mediawiki.org/wiki/Manual:Short_URL
  RewriteRule ^/*\$ %{DOCUMENT_ROOT}/w/index.php [L]
EOF
  find -L $(readlink -f $mw/..) -name .htaccess \
      | while read line; do
      echo -e "<Directory ${line%/.htaccess}>\n $(< $line)\n</Directory>";
  done
} | apache-site -i -p 127.0.0.1:8082 -- - $mwdomain
nginx-site -p 8082 $mwdomain
</source>


Turn on the new configuration
<source lang="bash">
if isdeb; then
    a2ensite $mwdomain.conf
    # for short urls
    a2enmod rewrite
    # We restart rather than reload because of the module
    service apache2 restart
else
    systemctl reload httpd.service
fi
</source>

Now mediawiki should load in your browser at $mwdomain .

Allow proper search bots and internet archiver bots, via [[Mediawiki:Robots.txt]]

<source lang="bash">
dd of=$mw/../robots.txt <<'EOF'
        User-agent: *
Disallow: /w/
        User-agent: ia_archiver
Allow: /*&action=raw
EOF
</source>

'''Skippable Notes'''

This section assumes we are redirecting www to a url without www.

[http://httpd.apache.org/docs/current/howto/htaccess.html Apache recommends] moving .htaccess rules into it's config for performance. So we look for .htaccess files from mediawiki and piwik and copy their contents into this config. In modern apache versions, we would have to explicitly set options like AllowOverride to allow .htaccess files to take effect.

== Mediawiki Settings ==

Overall reference: [[mediawikiwiki:Manual:Configuration_settings]].

Settings which the gui setup prompts for but aren't set by the automated install script.
<source lang="bash">
teeu $mwc<<EOF
\$wgServer = "https://$mwdomain";
\$wgDBserver = "localhost";
\$wgRightsUrl = "$mw_RightsUrl";
\$wgRightsText = "$mw_RightsText";
\$wgRightsIcon = "$mw_RightsIcon";
wfLoadSkin( 'Vector' );
EOF
</source>
Settings I recommend which are different than the defaults.
<source lang="bash">
teeu $mwc<<EOF
\$wgPasswordSender = "$mw_email";
\$wgEmergencyContact = "$mw_email";
\$wgEnotifUserTalk = true; # UPO
\$wgEnotifWatchlist = true; # UPO
\$wgMainCacheType = CACHE_ACCEL;
\$wgEnableUploads = true;
\$wgUseInstantCommons = true;
EOF
</source>

Other misc settings
<source lang="bash">
teeu $mwc <<'EOF'
# from https://www.mediawiki.org/wiki/Manual:Short_URL
$wgArticlePath = "/wiki/$1";

# https://www.mediawiki.org/wiki/Manual:Combating_spam
# check that url if our precautions don't work
# not using nofollow is good practice, as long as we avoid spam.
$wgNoFollowLinks = false;
# Allow user customization.
$wgAllowUserJs = true;
$wgAllowUserCss = true;

# use imagemagick over GD
$wgUseImageMagick = true;
EOF


# https://www.mediawiki.org/wiki/Manual:Configuring_file_uploads
# Increase from default of 2M to 100M.
# This will at least allow high res pics etc.
php_ini=$(isdeb && echo /etc/php5/apache2/php.ini || echo /etc/php.ini)
sed -i 's/^\(upload_max_filesize\)\b.*/\1 = 100M/'
sed -i 's/^\(post_max_size\)\b.*/\1 = 100M/'
if isdeb; then
    service apache2 restart
else
    systemctl restart httpd.service
fi

# if you were to install as a normal user, you would need this for images
# sudo usermod -aG $apache_user $USER

# this doesn't propogate right away
chgrp -R $apache_user $mw/images
chmod -R g+w $mw/images
</source>

Style settings. Omit to use a different style.
<source lang="bash">
teeu $mwc <<'EOF'
$wgLogo = null;
$wgShowIPinHeader = false;
$wgFooterIcons = null;
EOF
# Make the toolbox go into the drop down.
cd $mw
git remote add ian git@gitorious.org:mediawiki-toolbox-patch/mediawiki-toolbox-patch.git
git fetch ian
git rebase ian/REL1_23-toolbox-in-dropdown
</source>

== Install and Configure Mediawiki Extensions ==

When installing extensions on a wiki with important content, backup first as a precaution.

''' Extensions with no configuration needed '''

{| class="wikitable"
! Name
! Description
|-
| [[mediawikiwiki:Extension:Cite|Extension:Cite]]
| Have references in footnotes.
|-
| [[mediawikiwiki:Extension:CiteThisPage|Extension:CiteThisPage]]
| Ability to generate citations to pages in a variety of styles.
|-
| [[mediawikiwiki:Extension:CSS|Extension:CSS]]
| Allows CSS stylesheets to be included in specific articles
|-
| [[mediawikiwiki:Extension:DynamicPageList|Extension:DynamicPageList]]
| Embed page lists from categories & combinations of categories
|-
| [[mediawikiwiki:Extension:Echo|Extension:Echo]]
| Notification subsystem for usage by other extensions
|-
| [[mediawikiwiki:Extension:Gadgets|Extension:Gadgets]]
| UI extension system for users
|-
| [[mediawikiwiki:Extension:ImageMap|Extension:ImageMap]]
| Links for a region of an image
|-
| [[mediawikiwiki:Extension:Interwiki|Extension:Interwiki]]
| Tool for nice links to other wikis
|-
| [[mediawikiwiki:Extension:News|Extension:News]]
| Embed or rss recent changes
|-
| [[mediawikiwiki:Extension:Nuke|Extension:Nuke]]
| Mass delete of pages, in the case of spam
|-
| [[mediawikiwiki:Extension:ParserFunctions|Extension:ParserFunctions]]
| Useful for templates
|-
| [[mediawikiwiki:Extension:Poem|Extension:Poem]]
| Useful for formatting things various ways
|-
| [[mediawikiwiki:Extension:SyntaxHighlight_GeSHi|Extension:SyntaxHighlight_GeSHi]]
| Source code highlighting
|-
| [[mediawikiwiki:Extension:Variables|Extension:Variables]]
| Define per-page variables
|}

<source lang="bash">
mw-ext Cite CiteThisPage CSS DynamicPageList Echo Gadgets ImageMap Interwiki News \
       Nuke ParserFunctions Poem SyntaxHighlight_GeSHi Variables
</source>


''' [[mediawikiwiki:Extension:AntiSpoof|Extension:AntiSpoof]]: Disallow usernames with unicode trickery to look like existing names'''

<source lang="bash">
mw-ext AntiSpoof
# recommended setup script to account for existing users
sudo -u $apache_user php $mw/extensions/AntiSpoof/maintenance/batchAntiSpoof.php
</source>


''' [[mediawikiwiki:CheckUser|Extension:CheckUser]]: Get ip addresses from inside mediawiki so you can ban users'''

Requires special install steps or we can get into a bad state. Add a sleep like the default of update.php to avoid errors.
<source lang="bash">
mw-ext CheckUser
sudo -u $apache_user php $mw/extensions/CheckUser/install.php; sleep 1
</source>


'''[[mediawikiwiki:Extension:Wikidiff2|Extension:Wikidiff2]]: Faster and international character supported page diffs'''

I used packaged version since this is a c++ and probably not very tied to the Mediawiki version. This isn't packaged in fedora, haven't gotten around to testing and adding the code to compile it for fedora.
<source lang="bash">
if isdeb; then
    apt-get -y install php-wikidiff2
    teeu $mwc <<'EOF'
$wgExternalDiffEngine = 'wikidiff2';
EOF
    ln -sf ../../mods-available/wikidiff2.ini /etc/php5/apache2/conf.d
    service apache2 restart
fi
</source>


''' [[mediawikiwiki:Extension:Math|Extension:Math]] Display equations'''

<source lang="bash">
mw-ext Math
# php5-curl according to Math readme
if isdeb; then
    apt-get -y install latex-cjk-all texlive-latex-extra texlive-latex-base ghostscript imagemagick ocaml php5-curl
else
    # todo, php5-curl equivalent on fedora
    yum -y install texlive-cjk ghostscript ImageMagick texlive ocaml
fi
ln -sf ../../mods-available/curl.ini /etc/php5/apache2/conf.d
service apache2 restart

cd $mw/extensions/Math/math; make # makes texvc
cd $mw/extensions/Math/texvccheck; make

teeu $mwc <<'EOF'
# Enable MathJax as rendering option
$wgUseMathJax = true;
# Enable LaTeXML as rendering option
$wgMathValidModes[] = 'latexml';
# Set LaTeXML as default rendering option, because it is nicest
$wgDefaultUserOptions['math'] = 'latexml';
EOF
</source>

'''Skippable notes'''

There is no current list of package depencies so I took dependencies from mediawiki-math package in Debian 7. Fedora didn't have a mediawik math package, so I just translated from debian. Ocaml is for math png rendering, as backup option to the nicer looking LatexML and MathJax. Debian has texvc package, but it didn't work right for me, plus it required additional configuration in mediawiki settings.


''' [[mediawikiwiki:Extension:SpamBlacklist|Extension:SpamBlacklist]]: Import/create IP blacklists, mainly for spam'''

<source lang="bash">
mw-ext SpamBlacklist
if ! grep -F '$wgSpamBlacklistFiles = array(' $mwc &>/dev/null; then
    tee -a $mwc <<'EOF'
$wgEnableDnsBlacklist = true;
$wgDnsBlacklistUrls = array( 'xbl.spamhaus.org', 'dnsbl.tornevall.org' );

ini_set( 'pcre.backtrack_limit', '10M' );
$wgSpamBlacklistFiles = array(
   "[[m:Spam blacklist]]",
   "http://en.wikipedia.org/wiki/MediaWiki:Spam-blacklist"
);
EOF
fi
</source>

''' [[mediawikiwiki:Extension:TitleBlacklist|Extension:TitleBlacklist]]: Anti-spam '''

<source lang="bash">
mw-ext TitleBlacklist
if ! grep -F '$wgTitleBlacklistSources = array(' $mwc &>/dev/null; then
    tee -a $mwc <<'EOF'
$wgTitleBlacklistSources = array(
    array(
         'type' => 'local',
         'src'  => 'MediaWiki:Titleblacklist',
    ),
    array(
         'type' => 'url',
         'src'  => 'http://meta.wikimedia.org/w/index.php?title=Title_blacklist&action=raw',
    ),
);
EOF
fi
</source>

''' [[mediawikiwiki:Extension:WikiEditor|Extension:WikiEditor]]: Editing box extras and a fast preview tab '''

<source lang="bash">
mw-ext WikiEditor
teeu $mwc <<'EOF'
# Enable Wikieditor by default
$wgDefaultUserOptions['usebetatoolbar'] = 1;
$wgDefaultUserOptions['usebetatoolbar-cgd'] = 1;

# Display the Preview and Changes tabs
$wgDefaultUserOptions['wikieditor-preview'] = 1;
EOF
</source>

''' [[mediawikiwiki:CategoryTree|Extension:CategoryTree]]: Enables making nice outlines of pages in a category'''
<source lang="bash">
mw-ext CategoryTree
teeu $mwc <<'EOF'
# Mediawiki setting dependency for CategoryTree
$wgUseAjax = true;
EOF
</source>

''' [[mediawikiwiki:Extension:AbuseFilter|Extension:AbuseFilter]]: Complex abilities to stop abuse '''

Used by big wiki sites. As a smaller site, we won't use it much, but it's good to have. It's page suggests a few defaults:
<source lang="bash">
mw-ext AbuseFilter
teeu $mwc<<'EOF'
$wgGroupPermissions['sysop']['abusefilter-modify'] = true;
$wgGroupPermissions['*']['abusefilter-log-detail'] = true;
$wgGroupPermissions['*']['abusefilter-view'] = true;
$wgGroupPermissions['*']['abusefilter-log'] = true;
$wgGroupPermissions['sysop']['abusefilter-private'] = true;
$wgGroupPermissions['sysop']['abusefilter-modify-restricted'] = true;
$wgGroupPermissions['sysop']['abusefilter-revert'] = true;
EOF
</source>

'''[[mediawikiwiki:Extension:ConfirmEdit|Extension:ConfirmEdit]]: Custom Captcha'''

Uses captchaArray defined in mw_vars.

<source lang="bash">
mw-ext ConfirmEdit
captchaArray
teeu $mwc <<'EOF'
wfLoadExtension( 'ConfirmEdit/QuestyCaptcha' );
$wgCaptchaClass = 'QuestyCaptcha';
# only captcha on registration
$wgGroupPermissions['user'         ]['skipcaptcha'] = true;
$wgGroupPermissions['autoconfirmed']['skipcaptcha'] = true;
EOF
if ! grep -Fx 'foreach ( $localSettingsQuestyQuestions as $key => $value ) {' $mwc; then
    tee -a $mwc <<'EOF'
foreach ( $localSettingsQuestyQuestions as $key => $value ) {
        $wgCaptchaQuestions[] = array( 'question' => $key, 'answer' => $value );
}
EOF
fi
</source>

Enable account creation that we initially disabled.
<source lang="bash">
sed -i "/\\\$wgGroupPermissions\\['\\*'\\]\\['createaccount'\\] = false;/d" $mwc
</source>

== Additional Configuration with Pywikibot ==

There are quite a few [[mediawikiwiki:Help:Namespaces|special pages]] which act like variables to configure special wiki content and style. A big part of this wiki's style is configured in this section. We use Pywikibot to automate editing those pages.


'''Pywikibot Install'''

[[mediawikiwiki:Manual:Pywikibot/Installation|Manual:Pywikibot/Installation]]

<source lang="bash">
# get repo
mkdir ~/opt
git clone --recursive https://gerrit.wikimedia.org/r/pywikibot/core.git ~/opt/pywikibot
cd ~/opt/pywikibot
#updating
git pull --all
git submodule update
</source>


'''Pywikibot Configuration'''

Relevent docs: [[mediawikiwiki:Manual:Pywikibot/Use_on_non-WMF_wikis|Manual:Pywikibot/Use_on_non-WMF_wikis]],  [[mediawikiwiki:Manual:Pywikibot/Quick_Start_Guide|Manual:Pywikibot/Quick_Start_Guide]]


<source lang="bash">
cd $HOME/opt/pywikibot
dd of=user-config.py <<EOF
mylang = 'en'
usernames["$mwfamily"]['en'] = u'$wikiuser'
family = "$mwfamily"
console_encoding = 'utf-8'
password_file = "secretsfile"
EOF

dd of=secretsfile <<EOF
("$wikiuser", "$wikipass")
EOF

# it won't overrwrite an existing file. Remove if if one exists
rm -f  pywikibot/families/${mwfamily}_family.py
python generate_family_file.py https://$mwdomain/wiki/Main_Page "$mwfamily"

# Note, this needed only for ssl site
tee -a pywikibot/families/${mwfamily}_family.py<<'EOF'
    def protocol(self, code):
        return 'https'
EOF
</source>


'''Pywikibot Script'''

This will take a full minute or so because the bot waits a few seconds between edits. Useful doc: [[mediawikiwiki:Pywikipediabot/Create_your_own_script]].

<source lang="bash">
pw="$HOME/opt/pywikibot"

dd of=$pw/scripts/${mwfamily}_setup.py<<EOF
import pywikibot
import time
import sys
site = pywikibot.Site()
def x(p, t=""):
   page = pywikibot.Page(site, p)
   page.text = t
   #force is for some anti-bot thing, not necessary in my testing, but might as well include it
   page.save(force=True)

# Small/medium noncommercial wiki should be fine with no privacy policy
# based on https://www.mediawiki.org/wiki/Manual:Footer
x("MediaWiki:Privacy")

# licenses for uploads. Modified from the mediawiki's wiki
x("MediaWiki:Licenses", u"""* Same as this wiki's text (preferred)
** CC BY-SA or GFDL| Creative Commons Attribution ShareAlike or GNU Free Documentation License
* Others:
** Unknown_copyright|I don't know exactly
** PD|PD: public domain
** CC BY|Creative Commons Attribution
** CC BY-SA|Creative Commons Attribution ShareAlike
** GFDL|GFDL: GNU Free Documentation License
** GPL|GPL: GNU General Public License
** LGPL|LGPL: GNU Lesser General Public License""")
x("MediaWiki:Copyright", '$mw_license')
x("MediaWiki:Mainpage-description", "$mwdescription")



# The rest of the settings are for the site style

# Remove various clutter
x("MediaWiki:Lastmodifiedat")
x("MediaWiki:Disclaimers")
x("MediaWiki:Viewcount")
x("MediaWiki:Aboutsite")
# remove these lines from sidebar
# ** recentchanges-url|recentchanges
# ** randompage-url|randompage
# ** helppage|help
x("MediaWiki:Sidebar", """* navigation
** mainpage|mainpage-description
* SEARCH
* TOOLBOX
* LANGUAGES""")

# remove side panel
# helpfull doc: https://www.mediawiki.org/wiki/Manual:Interface/Sidebar
x("mediawiki:Common.css", """/* adjust sidebar to just be home link and up top  */
/* panel width increased to fit full wiki name. */
div#mw-panel { top: 10px; padding-top: 0em; width: 20em }
div#footer, #mw-head-base, div#content { margin-left: 1em; }
#left-navigation { margin-left: 1em; }


/* logo, and toolbar hidden */
#p-logo, #p-tb.portal {
   display:none;
}

/* make the font size smaller for the misc stuff */
#p-personal {
   font-size: 0.8em;
}

#footer-info {
   font-size: 0.8em;
}
div#mw-content-text {
    max-width: 720px;
}
""")
EOF

cd $pw
python pwb.py ${mwfamily}_setup
</source>


''' Skippable Notes '''

The docs suggest manually entering the pass with python pwb.py login.py, then it should stay logged in. That didn't work for me, and anyways, we automation, so we use secrets file method.

Family name, and all its duplicattions documented as supposed to be $wgSitename, but it works fine using any name.

== Automatic Backups ==

Copy this to a file (the next block assumes ~/bin/remote_wiki_backup) on a different machine than the one hosting mediawiki. Also copy ~/mw_vars to the same machine. Setup passwordless sudo to the mediawiki host (specifically, name the host $mwdomain). This will make a versioned backup of the wiki to ~/backup/wiki_file_backup and ~/backup/wiki_db_backup.

<source lang="bash">
#!/bin/bash
source ~/mw_vars

# if we get an error, keep going but return it at the end
last_error=0
trap 'last_error=$?' ERR

ssh root@$mwdomain <<ENDSSH
tee -a $mwc<<'EOF'
$wgReadOnly = 'Dumping Database, Access will be restored shortly';
EOF
mysqldump -p$dbpass --default-character-set=binary my_wiki  > ~/wiki_backup/wikidump
sed -i '$ d' $mwc # delete last line
ENDSSH
rdiff-backup root@$mwdomain::/root/wiki_db_backup ~/backup/wiki_db_backup
rdiff-backup root@$mwdomain::$mw ~/backup/wiki_file_backup

exit $last_error
</source>

Run this script on the backup machine to setup daily backups at 4 am.

<source lang="bash">
chmod +x $HOME/bin/remote_wiki_backup
x="$(mktemp)"
crontab -l > "$x"
teeu "$x" <<'EOF'
0 4 * * * x=$($HOME/bin/remote_wiki_backup 2>&1); [[ $? != 0 ]] && echo "$x"
EOF
crontab "$x"
</source>

If you are like most people and don't use the old-school mail spool, setup the backup system to send mail externally. Some ways to do that are on this [http://unix.stackexchange.com/questions/36982/can-i-set-up-system-mail-to-use-an-external-smtp-server stackoverflow answer]. Then make local mail to your user get forwarded to an address you will read:

'''Optional & requires additional steps'''
<source lang="bash" type="example">
sed -i "/^root:/d" /etc/aliases
echo "root: EXAMPLE_ONLY_REPLACE_ME@gmail.com" >> /etc/aliases
newaliases
</source>

== Restoring Backups ==

'''Whenever you implement a backup system, you should test that restoring the backup works.'''

You ''should'' be able to restore your wiki to a new machine by repeating all install steps, then restoring the database and the images directory. However, we backup the entire Mediawiki directory in case you forget to record a step, or some misbehaving code stores some state in a file. Since most people don't record the steps they took to setup Mediawiki, this is also the officially recommended method. Here we restore only the database and images folder, which should help identify any of those aforementioned issues. See [[mediawikiwiki:Manual:Restoring a wiki from backup]] if you run into any problems.

To test a backup restore:
# Do a backup of your wiki with some content in it, as described in the previous section
# Move your mediawiki install directory, or setup Mediawiki on a new machine
# Re-execute the mediawiki install steps
# Change REPLACE_ME in the code below (as in the backup section so you get the right variables),
# Execute the code on the backup machine.

<source lang="bash">
source ~/mw_vars
HOSTNAME=REPLACE_ME source ~/mw_vars
rdiff-backup -r now ~/backup/wiki_file_backup /tmp/wiki_file_restore
scp -r /tmp/wiki_file_restore/images root@$mwdomain:$mw/images
rdiff-backup -r now ~/backup/wiki_db_backup /tmp/wiki_db_restore
scp -r /tmp/wiki_db_restore root@$mwdomain:/tmp
ssh root@$mwdomain "mysql -u root -p$dbpass my_wiki < /tmp/wiki_db_restore/wiki_db_dump"
</source>

Then browse to your wiki and see if everything appears to work.

== Updates ==

Subscribe to get release and security announcements [https://lists.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki-announce].

For updates, we simply git pull all the repos, then run the maintenance script. This should be done after a backup. We recommend automatic updates to get security fixes and since not much is changing on the release branch. In this example, we update at 5 am daily (1 hour after the automatic backup example).

Major version upgrades should be done manually, and it is recommended to use a new installation directory and the same procedure as for backup & restore. Official reference: [[mediawikiwiki:Manual:Upgrading|Manual:Upgrading]]

Minor updates script:
<source lang="bash">
s=$HOME/bin/mediawiki_update
dd of=$s<<'EOF'
#!/bin/bash
source ~/mw_vars
cd $mw
git fetch --all
git checkout origin/$mw_branch
git rebase ian/REL1_23-toolbox-in-dropdown
cd extensions
for x in *; do
    if [[ -d $x ]]; then
        cd $x
        git fetch --all
        git checkout origin/$mw_branch || git checkout -qf origin/master
        cd ..
    fi
done
php $mw/maintenance/update.php -q
EOF
chmod +x $s

x="$(mktemp)"
crontab -l > "$x"
teeu "$x" <<EOF
0 5 * * * $s
EOF
crontab "$x"
</source>

== Upgrading Major Versions ==

Reference documentation is at [[mediawikiwiki:Manual:Upgrading]]

My strategy is:

# Read the "Upgrade notices for MediaWiki administrators" on the upgrade version and any skipped versions at [[mediawikiwiki:Version_lifecycle]].
# Setup a blank test wiki with the new version.
# Backup the old database, restore it to the new wiki, run php maintenance/update.php.
# If everything looks good, repeat and replace the old wiki with the new one.

== Setup Piwik ==

<source lang="bash">
# docs suggests using separate database user, google lead me here for how
# https://www.digitalocean.com/community/tutorials/how-to-create-a-new-user-and-grant-permissions-in-mysql
mysql -u root -p$dbpass <<EOF
CREATE USER 'piwik'@'localhost' IDENTIFIED BY '$piwik_pass';
GRANT ALL PRIVILEGES ON piwik . * TO 'piwik'@'localhost';
FLUSH PRIVILEGES;
exit
EOF
</source>


''' Manual Steps'''

Unfortunately, Piwik does not document how to do a fully automated install, and although it certainly seems possible, my first glance at the config file after doing manual steps showed a lot of noise, so we are sticking with the manual steps for now.

In a browser, navigate to your_domain/analytics. Follow these steps [http://piwik.org/docs/installation/#the-5-minute-piwik-installation the-5-minute-piwik-installation], using the database password from the previous section.


We are going to use an extension to handle the javaScript tracking tag part.

Under settings -> user settings, click the cookie to exclude your current browser.


''' [[mediawikiwiki:Extension:Piwik Integration|Extension:Piwik Integration]] '''

<source lang="bash">
git clone https://github.com/DaSchTour/piwik-mediawiki-extension.git $mw/extensions/Piwik
mw-ext -g Piwik
teeu $mwc <<EOF
\$wgPiwikURL = '$mwdomain/analytics/';
\$wgPiwikIDSite = '1';
EOF
</source>


'''If this is not the first site you setup'''

You will need to adjust the previous $wgPiwikIDSite variable. The correct value can be found in the Piwik javaScript tracking tag code, in the line that looks like <code>_paq.push(['setSiteId', 2]);</code>

'''Automatic Updates'''

Automatically update on tuesday, 3 am server time. This code will create an update script at ~/bin/piwik-remote-update. Based on [http://piwik.org/docs/update/ piwik.org/docs/update].

<source lang="bash">
s=~/bin/piwik-update
mkdir -p ${s%/*}
dd of=$s <<EOF
#!/bin/bash
piwik_path=$mw/../analytics
cd "\$(mktemp -d)"
wget -q http://builds.piwik.org/piwik.zip
unzip -q piwik.zip
rm -f piwik.zip
cp \$piwik_path/config/config.ini.php piwik/config
cp -rf piwik/* \$piwik_path
# prevent making an email out of the standard success response
x="\$( \$piwik_path/console core:update )"
if [[ \$x != "Everything is already up to date." ]]; then
echo "\$x"
fi
EOF
chmod +x $s

x="$(mktemp)"
crontab -l > "$x"
teeu "$x" <<< "0 3 * * tue $s"
crontab "$x"
</source>


''' Skippable Notes '''

Piwik docs say use a subdir or a subdomain. googling finds a page which suggests using a subdomain, for more logical separation https://library.linode.com/web-applications/analytics/piwik/centos-5. However, mediawiki doesn't use the root dir of the web server, so the analytics folder can't mess things up by existing there, and it doesn't have any crazy virtualhost settings which might conflict, so there is no need. Plus we get ssl for free by sharing the same domain.

The docs also say: "When you have finished configuring Piwik, you can set more restrictive permissions (ie. read only) to the “piwik/config/” folder. Piwik will always write its data inside the “piwik/tmp/” folder, so this is the only folder that requires write permissions."

But since the files are already owned by the web server user, and already can't be written by anyone else, it seems a bit of a pointless half measure since the owner can override any permissions. Enabling the .htaccess files seems more useful, which it doesn't even mention.

We could also set up a backup for piwik data, but traffic stat data is not that important so I haven't bothered.

== Choosing Extensions ==

Mediawiki.org has pages for ~5200 extensions. Mediawiki maintains ~700 extensions [http://git.wikimedia.org/tree/mediawiki%2Fextensions.git in it's git repo]. Wikipedia uses [https://en.wikipedia.org/wiki/Special:Version over 100 extensions]. Major distributors package [[mediawikiwiki:Comparison_of_extensions_in_distributions| ~36 extensions]]. We looked closely at the distributor's and briefly at the Mediawiki repo extensions. We haven't found any other useful list or recommendations.

Here are brief descriptions of extensions that are part of distributions and why they were rejected for this wiki.

{| class="wikitable"
|+
| '''InputBox''' || Add html forms to pages. Can't imagine using it. Would install if I did.
|+
| '''Pdfhandler''' || Gallery of pages from a pdf file. Can't imagine using it. Would install if I did.
|+
| '''Footnote''' || deprecated in newer versions
|+
| '''NewUserNotif''' || Send me a notification when a user registers. Seems like an excessive notification.
|+
| '''NewestPages''' ||  A page creation history that doesn't expire like recent-changes. Meh
|+
| '''RSSReader''' || Embed an rss feed. Can't imagine using it. Would install if I did.
|+
| '''Openid''' || Poor UI. 2 pages & 2 links <login> <login with openid> which is confusing & ugly.
|+
| '''Validator''' || dependency of of semantic
|+
| '''Semantic''' || Seems like a lot of trouble around analyzing kinds of data which my wiki will not have.
|+
| '''wikicalendar''' || Make a calendar of events etc. Can't imagine using it. Would install if I did.
|}

== Misc Notes ==

''' Mediawiki Documentation Quality '''

Overall the documentation is good, but like wikipedia, it depends.

The closer a topic is to core functionality and commonly used features, the better the documentation is likely to be. My guess is that Wikimedia Foundation (WMF) has a competing priority of being a good upstream to mediawiki users and being good for their own sites. That, plus the multitude of unconnected extension developers, and official documentation is sometimes neglected in favor of bug reports, readme files, comments, code, and unpublished knowledge. User's edits vary in quality, and often aren't reviewed by anyone. If you run into an issue, try viewing/diffing the most recent version of a page by the last few editors.

One issue is that mediawiki.org needs a lot of organizing, deleting, and verifying of material, and that is relatively unpopular, tedious, and sometimes difficult work. The discussion pages of mediawiki.org are a wasteland of unanswered questions and outdated conversations, which is [https://www.mediawiki.org/wiki/Help:Talk_pages poor form] for a wiki. However, if you communicate well, you can get great help from their  [https://www.mediawiki.org/wiki/Communication support forum, irc, and mailing list].


'''Bash here documents, EOF vs 'EOF' '''

Here documents are used throughout this page, some people may not be aware of a small but important syntax. When the delimiter is quoted, as in <<'EOF', then the contents of the here document are exactly verbatim. Otherwise $ and ` are expanded as in bash, and must be quoted by \, which itself must then also be quoted to be used literally.


''' Mediawiki automation tools survey 7/2014 '''

Barely maintained:
* https://github.com/ianweller/mw
* http://search.cpan.org/~markj/WWW-Mediawiki-Client/bin/mvs
* https://github.com/alexz-enwp/wikitools 3000 lines of code, no response to a bug reports in 2/2014

Getting basic maintenance
* https://github.com/mwclient/mwclient 2000 lines of code

Actively developed, used by wikimedia foundation a lot.
* [[mediawikiwiki:Manual:Pywikibot]]


''' Troubleshooting Errors '''

If mediawiki fails to load, or shows an error in the browser, enable some settings and it will print much more useful information. [[mediawikiwiki:Manual:How to debug]]

''' License '''

This page and this wiki is licensed under cc-by-sa 4.0.
This means the code is compatible with gplv3.

== todo list for this page ==

* Evaluate any extensions bundled in 27 which I haven't checked out before.
* Update style patch link.
* Visual editor
* Upgrade instructions
* Consider changing default signature to not have talk link