Academic Markdown - writing scientific articles and a whole thesis in markdown

After some very unpleasant experiences of collaboratively writing papers, proposals or my thesis either in Microsoft Word, Google Docs or (previously WriteLatex), I tried to give authoring in markdown another shot. All of my documentation I already do in markdown (in Vim) on a daily basis. And whenever required I can just produce beautiful PDF documents or HTML versions of READMEs or notes. So why not use this very legible format for bigger documents?

A good primer can be found here:

Pandoc is pretty much required for this to work at the moment, because delivers the best ecosystem of markup extensions, filters, documentation and input/output formats. It is written in Haskell and thus a hefty installation of the Haskell compiler is required before installing Pandoc itself in most cases (my old laptop did not have enough memory to compile the package with default configurations) and Haskells nice package manager Cabal itself was kind of hard to install on older systems because of missing dependencies.

My make command for markdown files in Vim is now:

autocmd BufRead,BufNewFile *.md setlocal filetype=markdown spelllang=en_us spell makeprg=pandoc\ --filter\ pandoc-citeproc\ -o\ %<.pdf\ %

and I plan to include the following filters:

--filter pandoc-fignos \
--filter pandoc-eqnos \
--filter pandoc-tablenos \

but should really define this on a document basis in the meta-header with panflute.

Also, I will try to implement my own acronym/abbreviation filter that replaces the first occurrence of things like 'ncRNAs' with 'non-coding RNAs' if defined like ~ncRNA: non-coding RNA.

I will add to this blog post as my experience progresses.

→

2018/01/04 15:20 · thoelken

Open science platforms

I stumbled upon the following projects for creating open (or closed) science projects for collaborative work:

2017/09/28 14:56 · thoelken

How to make movies with VMD, imagemagick and ffmpeg

Export frames in VMD after the molecule is setup and looking pretty. The following will produce 360 frames for a full rotation around the Y axis at fixed 720p resolution using Tachyon as external renderer. This must be installed first via sudo apt install tachyon. By default the renderer does not support JPEG output.

set i 0;
while {$i < 360} {
  rotate y by 1
  set d [format "%04d" $i]
  render Tachyon out.dat tachyon %s -res 1280 720 -aasamples 2 -mediumshade %s -format PNG -o out_$d.png
  incr i

Be sure to render in the resolution you want your GIF or movie to be in the end. Otherwise you loose quality or ffmpeg will not accept certain aspect ratios (you will have to pad with black pixels and you don't want that!)

Assembling the frames into a GIF with imagemagick:

convert -quality 5 -loop 0 -delay 5 out_%04d.png animation.gif

The quality option did not change my results visibly (no idea if this is working as intended). The -loop 0 parameter controls how often the animation is repeated. If your animation is large and you want to be nice to users of old PCs, limit this to a couple of times, because the CPU will have to flip through all the frames indefinitively with 0. Experiment with delay based on how many frames you have for the speed of the animation.

Creating a short movie from the same frames using ffmpeg:

ffmpeg -i out_%04d.png -vcodec libx264 -preset slow -crf 22 -r 15 -threads 0 video.mp4

-i out_%04d.png are your rendered frames with four digids and leading zeros. vcodec libx264 uses x264 for H.264 encoding (which has wide spread compatibility). -preset slow controls the encoding speed and results in better quality/compression ratio during encoding. -crf 22 is a measure of quality; 23 is an accepted default, with smaller values resulting in less loss of information and 51 decreases the image quality to a artifact riddled pulp. -r 15 controls the framerate; if you do not want to produce too many frames due to space or computing time reduce this value, to slow down your video. -threads 0 gives you all the CPUs you have available. And video.mp4 is evidently the generated output file.

2017/09/22 08:00 · thoelken

How to clean Text from HTML tags and UTF-8 symbols

The mailing list I use for the bike project I'm involved in, is set to forward text mode only and in some cases spits out unintelligible HTML/UTF gibberish. To clean up the mess, I found a quite robust converter under MailChimp.

2017/05/02 07:45 · thoelken

The magic of join

Often we want to compare or connect two lists and see what the overlap is like. While this is possible in spreadsheet software (Excel and alike) it is often overly complicated and does not produce the results we would like.

UNIX join to the rescue.

With the following example we can join two files easily:

 cat students.csv
 > #studentID,name,semester
 > 0,Peter,3
 > 1,Anna,2
 > 2,Sonja,7
 cat grades.csv
 > #studentID,course,grade
 > 2,Physics,89
 > 0,Math,40
 > 0,Physics,30
 join students.csv grades.csv
 > #studentID,name,semester,course,grade
 > 0,Peter,3,Math,40
 > 0,Peter,3,Physics,30
 > 2,Sonja,7,Physics,89

What happened to Anna? She is not in both files and thus omitted from the output. If we want to include all entries from one file we can do so.

 join -a 1 students.csv grades.csv
 > #studentID,name,semester,course,grade
 > 0,Peter,3,Math,40
 > 0,Peter,3,Physics,30
 > 1,Anna,2
 > 2,Sonja,7,Physics,89

Better! But if we want to use this table to sort by grade or course there are no entries for Anna. In fact most parsers will complain, that row 4 has less fields than the others. We can include empty fields with added separators with the auto output format.

 join -a 1 -o auto students.csv grades.csv
 > #studentID,name,semester,course,grade
 > 0,Peter,3,Math,40
 > 0,Peter,3,Physics,30
 > 1,Anna,2,,
 > 2,Sonja,7,Physics,89

These two added commas will save us a lot of headache down the line.

2017/04/06 12:04 · thoelken

Older entries >>