Wikimedia Technical Conference 2019

The Wikimedia Technical Conference 2019, Atlanta concluded last week. I attended this conference for the first time, and it was an enjoyable experience for me. The central theme of the conference was around developer productivity, and several sessions were held covering different aspects of this theme.

Personal Experience

This year’s conference was the first technical conference that I attended for Wikimedia. It was a very delightful experience for me.


I got to attend and participate in some very insightful sessions. I met with a lot of people from the foundation and volunteers, and it was great interacting with them.

Commons Android app Team’s Participation

Our team took part in the session where Best practices & useful methods for remote teams was discussed.

Maskaravivek [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)]

Here is the presentation that we prepared to explain the process that our team follows.

Takeaways

As most of our core team was also present at the conference, we decided to have a short discussion to incorporate our learnings from the various sessions we attended, into our team’s processes.


Notes from our team’s discussion at the conference

Here’s a summary of the points that we discussed:

  • Reevaluating the usage of Kotlin in the Commons app. As of now, the usage of Kotlin is limited to just unit test cases, and we are trying to evaluate if it would be good if it can be used in the main classes. Based on our discussion with other folks at the conference, we decided to restart the discussion and arrive at a consensus on it. You can participate in the discussion here:

https://github.com/commons-app/apps-android-commons/issues/747

  • We have decided to start monthly skill share sessions where we will pick a topic, and one of the team members will talk about it over a video call. Some of the issues that we have chosen for the first few sessions are MVP architecture, Kotlin, Unit testing, Dagger, and RxJava.
  • We also agreed upon introducing some social connections amongst the team members and volunteers. We will have virtual coffee sessions every month, where we will talk about everything except work. :). Apart from these virtual coffee sessions, we will also create a new social channel on Zulip, where we will talk about non-work related stuff.
  • We also discussed the need for focussing on integrated tests and UI tests for the app. So we have decided to make Espresso tests part of CI and also make it a priority to add more flows in the UI tests.
  • In the next few months, we will also focus on integrating structured data into the commons app. We will build upon the work done by Vanshika Arora and try to release structured data support as soon as possible for everyone.
  • Lastly, we discussed validating if our app supports all languages that are supported by Android and MediaWiki.

Conclusion

The event was quite delightful to me. Overall it was a very fruitful conference for our team, and we were able to deliberate on several issues, which are sometimes not possible to discuss remotely. Here’s the group picture of all the attendees of the conference.

Group picture of all participants of Wikimedia Technical Conference 2019

Hopefully, the changes that we have discussed will help in improving our team’s productivity and make the app a lot more delightful for the users.

Querying Wikimedia Commons DB to get User Statistics

Getting User Statistics on Wikimedia Commons

Wikimedia Commons is an online repository of free-use images, sounds, and other media files. It is a project of the Wikimedia Foundation.

If you are a Commons user or need to get statistics for a particular user, then you can query the commonswiki DB to get the results. In this post, I will introduce you to a few SQL queries to get various user statistics.

Get Count of File Uploads

You can check the number of file uploads by a user using the following query:

use commonswiki_p;
select
count(*)
from
logging_userindex
where
log_type = "upload"
and log_user =(
select
user_id
from
user
where
user_name = "Maskaravivek"
);

You can run the query on Quarry.

https://quarry.wmflabs.org/query/31757

Get Count of Deleted file uploads

Commons has a strict deletion policy and the community can nominate a picture for deletion if your upload violates any of the policies.

You can use the following query to get the count of deleted file uploads.

select count(*) from commonswiki_p.filearchive_userindex where fa_user_text="Maskaravivek"

commonswiki_p.filearchive_userindex has an index on username and querying it is much faster than querying the original table ie. commonswiki_p.filearchive .

You can run the query on Quarry.

https://quarry.wmflabs.org/query/31757

Get Count of Articles Using Images

The whole purpose of the Commons image repository is to let Wikipedia articles use the images.

The query below gives the count of articles using images from a particular user.

select
count(*) as articlesUsing
from
commonswiki_p.globalimagelinks
where
gil_to in (
select
log_title
from
commonswiki_p.logging_userindex
where
log_type = "upload"
and log_user =(
select
user_id
from
commonswiki_p.user
where
user_name = "Maskaravivek"
)
);

You can run the query on Quarry.

https://quarry.wmflabs.org/query/31757

Get Count of Unique Images USed

The query below gives the count of articles using images from a particular user.

select
count(distinct gil_to) as uniqueUsed
from
commonswiki_p.globalimagelinks
where
gil_to in (
select
log_title
from
commonswiki_p.logging_userindex
where
log_type = "upload"
and log_user =(
select
user_id
from
commonswiki_p.user
where
user_name = "Maskaravivek"
)
);

You can run the query on Quarry.

https://quarry.wmflabs.org/query/31757

Get Count of Images Edited by Someone else

The pictures that you upload to commons can be edited by some other user as well. The following query can be used to get the count of such images.

use commonswiki_p;
select
count(*)
from
revision
where
rev_page in (
select
log_page
from
logging_userindex
where
log_type = "upload"
and log_user =(
select
user_id
from
user
where
user_name = "Maskaravivek"
)
)
and rev_user !=(
select
user_id
from
user
where
user_name = "Maskaravivek"
)
group by
rev_page
having
count(*) > 1

You can try the query on Quarry.

https://quarry.wmflabs.org/query/31757

Get Number of Thanks received

Users can express thanks to other users using Commons.

The following query can be used to get the count of the number of thanks received by a user.

use commonswiki_p;
select count(*) from logging_logindex where log_type="thanks" and log_title="Maskaravivek";

https://quarry.wmflabs.org/query/31757

The Commons Android app shows these statistics in a very nice interface.

User Achievements

Make sure you give this post 50 claps and follow me if you enjoyed this post and want to see more!

How to get started with Open Source?

Open Source by Nick Youngson CC BY-SA 3.0 Alpha Stock Images

There are a lot of developer and wiki guides to set up and get started with your first open source contribution. As a beginner, I always found these wikis a bit overwhelming as they suddenly introduce you to a lot of options and projects. As a beginner, I wasn’t looking for lots of options but rather wanted someone to hand hold me and help me find one project which was easy to start with open source.

Throughout my college days, open source fascinated me and I always wanted to contribute to some open source project but couldn’t find the right project. When I was in the second year of college I made a failed attempt to kick start my open source journey by applying for Google Summer of Code. I attribute my failure to these reasons:

  • I simply wanted to land an internship with GSoC and could never zero down on a suitable project to get started.
  • I found the developer guides overwhelming as they gave me a lot of options and I wasn’t able to decide what was best for me.
  • I didn’t dedicate enough time to methodically get started with open source. Rather than focus on finding the right project, I focussed on making the first patch and submitting a proposal.

It’s also very important to ask yourself why you want to contribute to open source before actually making a contribution. The answer might be different for different individuals but it should be convincing enough for you to make the choice.

In this post, I will share some tips on how to choose a project and make your first contribution to open source.

Why contribute to open source?

  • Improve your existing coding/design skills
  • Meet people who are interested in similar things. Open source communities are more often than not very welcoming and you might soon form lifelong friendships through your participation be it in online chats or running into each other at conferences.
  • Personally, I believe that having a mentor and mentoring someone is very important to learn continuously. Open source can fulfill both these needs.
  • It’s empowering to make even a very tiny contribution. I have made 100s of contributions to open source but am most proud of my little contribution on Wikipedia where I added an image to Parle-G’s article from Wikimedia Common’s image repository.

How to choose a project?

There are lots of open source projects out there, but for a beginner, your options might be limited. Most of the popular projects on Github are large codebases which might be a bit overwhelming for a beginner.

Personally, my criteria for choosing a project is based on the following factors:

Active Project

At first glance, the pulse as shown on Github can be a good indicator to check whether the project is being actively maintained or not.

A look at the number of open/closes issues, and open/merged pull requests give a fair idea about whether the project is still actively maintained.

Beginner Friendly

It should be a beginner friendly project. Project’s wiki is a good indication of how welcome new-comers are to the project. Project maintainers have to make lots of efforts to make their projects appealing to new-comers. Github has a guide for project maintainers that helps them make their project beginner friendly.

Here’s how Wikimedia Commons app’s community profile looks . We made an effort to tick off most of the items in the checklist so that its easier to get started for newcomers.


Some projects are open source but are not beginner friendly and have just a core group of developers who make contributions. For eg., some projects already have a core team of full-time developers who are paid to maintain the project. Such a project will people assigned to most of the issues and newcomers might have limited opportunities to make impactful contributions to it.

Coding Language

It should be in a language that I am comfortable with, eg. Java, Python. Rather than getting familiar with an entirely new language to contribute to a project, it’s much simpler to choose a project which uses a language you already know. I make most of my contributions to Wikimedia Commons Android app as I am already familiar with Java and Android. It makes it much easier to understand the existing code base and make contributions to it.

Project Popularity

It should be reasonably popular. The number of stars/forks a project has can be a fair index to gauge the popularity. Naturally, a project like Linux with 780K commits and 62K stars would be overwhelming for a beginner. I would instead choose a project with a few thousand commits and 100–5000 stars. I believe the chances for it being beginner friendly is much more when compared with an immensely popular project.

Also, taking a look at other statistics like last commit, contributors, traffic, code frequency, etc. can give a fair idea about the project’s popularity and its activity.

Hosted on Github

Also, it should preferably be hosted on Github. If the project uses some other tool for code review, for instance, Gerrit then its a bit of turn off for me as it requires additional steps for setting up the environment. For eg., most of Wikimedia projects use Gerrit for code review. I never found it appealing enough to set it up and start using it. I agree that it’s not a significant factor while deciding on the project but I prefer Github for the entire workflow.


Conclusion

Once you have zeroed down on the project that you want to contribute to, the next step is finding the first issue that you can pick up and fix. Keep in mind that you don’t have to stick to the project forever if you don’t like it. The project is meant to make you get started with the journey. Open source guide has a few checklists that could help you to get started with your open source journey.

Here are some curated lists of beginner-friendly projects/issues:


Make sure you give this post 50 claps and follow me if you enjoyed this post and want to see more!

Fetching Featured Images from Wikimedia Commons

Recently featured image on Commons: https://bit.ly/2IyitSG

Wikimedia Commons features many images regularly on its main page. As described by Commons,

Featured pictures are images from highly skilled photographers and illustrators that the Wikimedia Commons community has chosen as some of the highest quality on the site.

Getting an image featured is quite difficult as the statistics suggest.

There are currently 11,192 of such images in the Commons repository which is roughly 0.024% of the available images (46,591,843).

In the Wikimedia Commons Android app, we thought showing these featured images in the app would be a good addition as it would:

  • Show users what a great Commons picture is
  • Inspire/motivate users to take part in the same project
  • Let users hope that one day their picture will be shown at that place (in fact, show them that taking great pictures is worth the effort)

I started exploring the API that would let us fetch featured images and then we could consume it in the app. I came across the MediaWiki Generator API.

Check out the following API call that returns the first 10 features images from the API. https://bit.ly/2KRXg4l

Here’s a screenshot from the app showing the featured images from Commons:

Featured Images Screen

Let us discuss the parameters that the API takes:

  • action: query which is common param for all MediaWiki API calls
  • generator: we are going to generate a list of categorymembers
  • gcmtype: this param’s value is file as we want to extract information of the Commons image files
  • gcmtitle: this is name of the category under which all featured images on Commons are stored. Category:Featured_pictures_on_Wikimedia_Commons
  • prop: Setting the value of prop to imageinfo adds the imageinfo block in the API response. It contains the url and extmetadata fields that are required for getting some of the properties.

We wanted to display all the images in a category instead of showing just the first 10. So we used the continue query of MediaWiki API. To get more data, you use the query-continue value in the response.

The query-continue node will contain a subnode for each module used in the query that needs continuation, and these subnodes will contain properties to be used when making the followup “continuation” query. Note that clients should not be depending on the particular property names given for continuation of any module or the format of the values returned for continuation, as these may change.

Continue query: https://bit.ly/2Idkhgt

Moreover, the implementation in generic enough to load images from any category of Commons.

CategoryImagesActivity.startYourself(this, "Page title", "Category name");

Check out the full implementation for this feature here:

https://github.com/commons-app/apps-android-commons/pull/1456

The changes are now merged to master and you could take a pull of the code base and run it for yourself. 🙂

https://github.com/commons-app/apps-android-commons/pull/1456


Make sure you give this post 50 claps and follow me if you enjoyed this post and want to see more!

Wikimedia Commons App’s Participation in FOSS Internship Programs

The Wikimedia Commons Android app is an open-source app created and maintained by grantees and volunteers of the Wikimedia community.

https://play.google.com/store/apps/details?id=fr.free.nrw.commons

The Wikimedia Foundation is not involved directly in the creation, development or creation of the app. For an app fully maintained by the non-paid community members, this pulse is an extraordinary feat!

Pulse for the last one month period

If you dig down to the contributor’s section for January to March period, you would see that most of the contributors are new to the project. This influx of new contributors is as we included our app for the FOSS internship programs for the following projects

We definitely expected a few students to come and show their interest to the app but the response was simply overwhelming. At least 10 new contributors made their first contribution to the app hoping to grab an internship opportunity.

We made our last release ie. v.2.6.7 on 1st Feb and for the next release, we have a number of features lined up, thanks to the students. Some of the notable feature enhancements added by students apart from bug/crash fixes are:

Here are some screenshots from the app showing what has changed.


Updated About page and Login Screen


Revamped Settings and Welcome Screens

We really appreciate the effort all the students have put in to identify issues and fix them. Thanks to them we have managed to fix several bugs and add small yet good to have features which would have otherwise remained in our backlog for a long time.

This is the first mentorship experience for me and I am really excited about interacting with my mentee. I had previously co-mentored Google Code In students for the same project and it really felt good helping school kids get started with their open source journey.

https://play.google.com/store/apps/details?id=fr.free.nrw.commons

Volunteers are always welcome to the project! Feel free to check out the project and make a contribution[ 😉 ] to the Commons Android app.

https://play.google.com/store/apps/details?id=fr.free.nrw.commons

If you want to get started with Open source you can read my post on how to choose the right project and submit your first pull request.

https://play.google.com/store/apps/details?id=fr.free.nrw.commons

SPARQL session at the Wikimedia Prague Pre-Hackathon

SPARQL Session at Prague Pre-hackathon

Am at the Prague pre-hackathon and the guys from Wikipedia UK and Wikipedia Austria gave a session on SPARQL queries. It seems to be a very cool way to get results from Wikidata and visualize it. Adam Shorland and Tobias Schönberg took this session for all the Wikimedia Commons App team in Wikimedia Czech Republic office.

They gave us an introduction to get started with SPARQL queries. SPARQL is an RDF query language, that is, a semantic query language for databases, able to retrieve and manipulate data stored in Resource Description Framework (RDF) format.

Here are a few sample queries that can give you a feeling of how it works.

Query to see a list of world heritage sites

SELECT ?item ?itemLabel ?coord ?image { ?item wdt:P1435 wd:Q9259 . ?item wdt:P17 ?country . ?item wdt:P625 ?coord . ?item wdt:P18 ?image SERVICE wikibase:label {bd:serviceParam wikibase:language "en"} }

See the results here. Here’s a sample output.

List of world heritage sites

Query to show a list of nearby places without any images

The Android Wikimedia Commons app uses this query to get a list of nearby places which don’t have an image on Wikimedia Commons. Anyone is welcome to contribute to the Android app.

SELECT
(SAMPLE(?location) as ?location)
?item
(SAMPLE(COALESCE(?item_label_preferred_language, ?item_label_any_language)) as ?label)
(SAMPLE(?classId) as ?class)
(SAMPLE(COALESCE(?class_label_preferred_language, ?class_label_any_language, "?")) as ?class_label)
(SAMPLE(COALESCE(?icon0, ?icon1)) as ?icon)
(SAMPLE(COALESCE(?emoji0, ?emoji1)) as ?emoji)
?wikipediaArticle
?commonsArticle
WHERE {
# Around given location...
SERVICE wikibase:around {
?item wdt:P625 ?location.
bd:serviceParam wikibase:center "Point(${LONG} ${LAT})"^^geo:wktLiteral.
bd:serviceParam wikibase:radius "${RAD}" . # Radius in kilometers.
}

# ... and without an image.
MINUS {?item wdt:P18 []}

# Get the label in the preferred language of the user, or any other language if no label is available in that language.
OPTIONAL {?item rdfs:label ?item_label_preferred_language. FILTER (lang(?item_label_preferred_language) = "${LANG}")}
OPTIONAL {?item rdfs:label ?item_label_any_language}

# Get the class label in the preferred language of the user, or any other language if no label is available in that language.
OPTIONAL {
?item p:P31/ps:P31 ?classId.
OPTIONAL {?classId rdfs:label ?class_label_preferred_language. FILTER (lang(?class_label_preferred_language) = "${LANG}")}
OPTIONAL {?classId rdfs:label ?class_label_any_language}

# Get icon
OPTIONAL { ?classId wdt:P2910 ?icon0. }
OPTIONAL { ?classId wdt:P279*/wdt:P2910 ?icon1. }
# Get emoji
OPTIONAL { ?classId wdt:P487 ?emoji0. }
OPTIONAL { ?classId wdt:P279*/wdt:P487 ?emoji1. }
OPTIONAL {
?sitelink schema:about ?item .
?sitelink schema:inLanguage "en"
}
OPTIONAL {
?wikipediaArticle schema:about ?item ;
schema:isPartOf <https://en.wikipedia.org/> .
SERVICE wikibase:label { bd:serviceParam wikibase:language "en" }
}

OPTIONAL {
?commonsArticle schema:about ?item ;
schema:isPartOf <https://commons.wikimedia.org/> .
SERVICE wikibase:label { bd:serviceParam wikibase:language "en" }
}
}
}
GROUP BY ?item ?wikipediaArticle ?commonsArticle

See the results here. Here’s a sample output.



Originally published at www.maskaravivek.com on May 13, 2017.

Bitnami