I think this is by design. Do not append .md to page names, just pick markdown in the dropdown Format menu, which btw is the default. You can use slugs with dashes (-), otherwise it gets escaped and produces an error.
However, I found a bug. If you do what you suggested:
go the pages tab and visit the newly created page you get a 404, because the url gets escaped and you see something like /wikis/dir%2Fanother_page. Replace %2F with a / and it should work.
root@localhost:/tmp/wiki-create-page-problem.wiki# ls -R
.:
dir home.markdown
./dir:
another_page.markdown page.md.markdown
root@localhost:/tmp/wiki-create-page-problem.wiki#
So, it seems that GitLab have problems with dots in page slugs and also forces the extension .markdown. This fact I can not see in the documentation.
As you also can see, the page.md was actually created as page.md.markdown, but the page can not be viewed via the GitLab web interface.
I do not have the problem viewing the another_page that you have. Klicking on the link opens the URL http://localhost/janne/wiki-create-page-problem/wikis/dir/another_page and shows allright.