Building a Links Log in Textpattern

   23 July 2004, late evening

Update Sept 13th 2005: The changes needed to publish.php are slightly different for Textpattern 4.0.1. Read my article on upgrading to 4.0.1 for more details. (There is a very concise post on how I did things in 4.0.1 for those who think this post is too all over the place.)

As I’ve mentioned before, I wanted to create a link log in a similar manner to Jason Kottke. What I wanted to accomplish were the following things:

  1. Links would be intermingled with posts. Everything would be displayed in chronological order.
  2. Since I was displaying a list of links, the links should semantically be marked up as unordered lists.
  3. I wanted to continue using Del.icio.us, as well as Textpattern.
  4. I wanted to import all my old entries from Del.icio.us into Textpattern.

I’ve managed to accomplish all of these goals. So, here are some details for those of you who enjoy this sort of thing.

Creating A Link Entry

The simplest thing to do was figure out how to store a link entry in my Textpattern system. Although Textpattern does have a feature for saving links, I opted to store links as articles. The primary reason for doing so is that I’d like people to be able to comment on links. A fellow who goes by Sencer outlined what I thought was a good way to save links as articles on the textpattern forms.

There are a few elements to a link entry that need to be saved. The link itself, a title, a description, keywords/tags, a referrer name, and a referrer url. There may be more things you may want to save, but I thought this was enough information for my needs. I mapped things as follows: the link title gets saved in the title field, the description in the article body, the url in custom field 1, the referrer in custom field 2, the referrer url in custom field 3, and the keywords/tags in the keywords field.

I wanted the link entries to display differently then the rest of my posts, so I created a new form to display them with. I have to set the override form to this new form. The links all end up in their own section, linklog.

Currently I don’t set categories for the links, though I could do so based on the keywords/tags I saved.

Posting a New Link

I created a simple web page with a form for entering the information I described above. The form takes the information entered, and by directly accessing the Textpattern database, saves a new article. Ideally one would want some sort of API for posting to Textpattern; unfortunately, no such API exists. After posting to Textpattern, the page will send of a request to Del.icio.us with the same information. In this way, I can post to both places simultaneously. I made a bookmarklet that copies the URL and title of the page I am viewing into the html page I made.

Displaying the Links on the Page

As I mentioned earlier, the links are just articles. Since the links have their display form overridden they will display intermingled with my posts by default. The challenge at this step was to get them to display as lists. This involved turning my Textpattern install into a bigger Frankenstien then it already is.

The override form for each link entry would print the link out as part of a list—i.e. it would nest the entry in <li> tags. What each link would need to do is determine is if it was the first or last link in a block of links. If it was, then it would need to start or end the list block—i.e. output a <ul> or </ul> tag.

Since each link is posted to its own section, I simply needed tags to check if the section for the previous article or the next article were different from the current articles section. If the section of an entry that proceeded a link was not the section links belong to, then we know that we are starting a block of links. Similarly, if the section of the entry following a link was not the section links belong to, then we know we are ending a block of links.

I wrote functions to figure out the above information. One hiccup with this approach is that since we only display a finite number of articles on each page, it may be the case we need to end, or start, a block of links even though the next, or proceeding, article’s section is the link log section. Links at the top or bottom of the page may suffer from this problem. To get around this, I added an other variable to each article that said which article on the page it was. If we were printing a link, and it was the first article on a page, then we would always need to start the list it would be placed in, even though on the previous page it may be the case that there was a link entry. Similarly, if we are printing out a link and it was the last entry in a page, we would need to end the list it is placed in.

This step probably took the most time. The lack of documentation in Textpattern can make these sort of endeavours difficult. However, it is usually instructive to mess around with someone else’s code.

Importing My Old Del.icio.us Entries

The last thing I needed to do was import all my entries from Del.icio.us into Textpattern. Now, thankfully, Del.icio.us is an excellent and very open system. I found a simple curl command for grabbing all your links on the web, and used it to grab all my links. (The command is curl -u user:password http://del.icio.us/api/posts/recent?count=99999 > d
el.icio.us-backup.xml
)

Now, I had a XML file full of links. Unfortunately, I don’t know how to parse files like that in PHP. I knew how to do so in Python, but didn’t know how to do database inserts in Python.

I began by trying to make a python script to generate the SQL commands for inserting all the links. My plan was to run them by hand. Simply copy and paste what the python script generated into phpMyAdmin and that would be that. But there were issues with un-escapped strings, and the fact I needed to textile the descriptions. So I scrapped that, and started writing a PHP script.

Now, instead of trying to learn how to parse XML in PHP, I decided I would use some simple regular expressions to turn the XML file into a PHP array. I then copied that array into a PHP program I wrote that would iterate through the array and perform SQL inserts. The advantage to going this route was that I could use a lot of the functions in Textpattern to do the database calls and format the information.

That script wasn’t too hard to write, and worked fine.

Conclusions

So there you have it. A poorly written account on how I spent the past day.

Apendix

I’ve decided to post some of the code I used to do all this here, since I’ve been asked for it on three different occasions now. If you make the changes outlined below, you can basically clone what I have done here.

I added two functions to the taghandlers.php file. They could be turned into plugins:

  1. function if_next_section_different($atts, $thing) {
  2. global $thisarticle, $pretext;
  3. if ($thisarticle['article_num'] == 1) return parse($thing);
  4. $posted = safe_field('posted','textpattern','id = '.$thisarticle['thisid']);
  5. $thenext = getNeighbour($posted,$pretext['s'],'>');
  6. return ($thisarticle['section'] != $thenext['section']) ? parse($thing) : '';
  7. }
  8.  
  9. function if_prev_section_different($atts, $thing) {
  10. global $thisarticle, $pretext, $limit;
  11. if ($thisarticle['article_num'] == $limit) return parse($thing);
  12. $posted = safe_field('posted','textpattern','id = '.$thisarticle['thisid']);
  13. $theprev = getNeighbour($posted,$pretext['s'],'<');
  14. return ($thisarticle['section'] != $theprev['section']) ? parse($thing) : '';
  15. }
  16. Download this code snippit /static/code/1.txt

I changed the getNeighbour() function in publish.php:

  1. function getNeighbour($Posted, $s, $type)
  2. {
  3. $type = ($type == '>') ? '>' : '<';
  4. $q = array(
  5. "select ID, Title, Section, url_title, unix_timestamp(Posted) as uposted
  6. from ".PFX."textpattern where Posted $type '".doSlash($Posted)."'",
  7. ($s!='' && $s!='default') ? "and Section = '".doSlash($s)."'" : filterFrontPage(),
  8. 'and Status=4 and Posted < now() order by Posted',
  9. ($type=='<') ? 'desc' : 'asc',
  10. 'limit 1'
  11. );
  12.  
  13. $out = getRow(join(' ',$q));
  14. return (is_array($out)) ? $out : '';
  15. }
  16. Download this code: /static/code/2.txt

I use this form template to post the links to my front page. I have to select it as an override form. Using conditionals you could probably get away with out doing that though.

  1. <txp:if_article_list>
  2. <txp:rsx_if_next_section_different>
  3. <div class="post">
  4. <ul class="interesting-link">
  5. <li class="first-child">
  6. <txp:else />
  7. <li>
  8. </txp:rsx_if_next_section_different>
  9. <txp:if_custom_field name="url"><a href="<txp:custom_field name="url" />"><txp:title /></a>&nbsp;<txp:body /><txp:else /><txp:body /></txp:if_custom_field><txp:if_custom_field name="referrer"> (via&nbsp;<a href="<txp:custom_field name="referrer url" />"><txp:custom_field name="referrer" /></a>)</txp:if_custom_field>&nbsp;&nbsp;<txp:comments_invite /><txp:rsx_frontend_edit_link prefix="&nbsp;&nbsp;[" suffix="]">Edit</txp:rsx_frontend_edit_link></li>
  10. <txp:rsx_if_prev_section_different>
  11. </ul>
  12. </div>
  13. </txp:rsx_if_prev_section_different>
  14. </txp:if_article_list>
  15.  
  16. <txp:if_individual_article>
  17. <div class="post">
  18. <h2><txp:if_custom_field name="url"><a href="<txp:custom_field name="url" />"><txp:title /> &rArr; </a><txp:else /><txp:permlink><txp:title /></txp:permlink></txp:if_custom_field></h2><p class="inline">&nbsp; &nbsp;<txp:posted />, <txp:rsx_time_of_day /></p>
  19. <p><txp:body /><txp:if_custom_field name="referrer"> This link was found via <a href="<txp:custom_field name="referrer url" />"><txp:custom_field name="referrer" /></a>.</txp:if_custom_field></p><p><span class="float_left"><txp:comments_invite /></span><span class="float_right"><txp:rsx_frontend_edit_link suffix=" | ">Edit</txp:rsx_frontend_edit_link><txp:permlink>Perma-Link</txp:permlink></span>&nbsp;</p>
  20. </div>
  21. </txp:if_individual_article>
  22.  
  23. Download this code: /static/code/3.txt

Also, and I forgot to email this out to some people who asked, you need to keep track of how many articles are being displayed on your page. The first article, if it is a link, should also start a link block, and the last article on a page, if it is a link, should always end a link block.

In the doArticles() function in publish.php, you will need to add an article number variable. So after the if ($rs) { and before the foreach($rs as $a) { add the line $article_num = 0;.

In the for loop, after the $author has been set, add this line: $out['article_num'] = ++$article_num;.

This is the code I use to actually post to both del.icio.us and textpattern. I use it in conjunction with a HTML page that supplies the required POST variables. You can see what values need to be supplied via a POST request in the source code below.

  1. <?php
  2. include 'config.php';
  3. define("txpath",$txpcfg['txpath']);
  4.  
  5. if (!file_exists('config.php')) die("There doesn't seem to be a config.php file. You must install Textpattern before you import any entries.");
  6.  
  7. include txpath.'/lib/admin_config.php';
  8. include txpath.'/lib/txplib_db.php';
  9. include txpath.'/lib/classTextile.php';
  10. include txpath.'/publish/taghandlers.php';
  11. include 'HttpClient.class.php';
  12.  
  13. $textile = new Textile;
  14.  
  15. // Set these values
  16. $author = 'user';
  17. $insert_with_status = 4;
  18. $insert_into_section = 'linklog';
  19. $override_form = 'Link-Post';
  20. $annotate = 1;
  21. $annotate_invite = '#';
  22. $textile_body = 0;
  23. $textile_excerpt = 0;
  24.  
  25. // Post variables
  26. $title = $_POST['description'];
  27. $body = $_POST['extended'];
  28. $body_html = $_POST['extended'];
  29. $keywords = $_POST['tags'];
  30. $link_url = $_POST['url'];
  31. $via = $_POST['via'];
  32. $via_url = $_POST['via_url'];
  33.  
  34. // set URL title
  35. $url_title = stripSpace($title);
  36.  
  37. // The Rest
  38. $posted_date = date('YmdHis');
  39.  
  40. // check the url title is set
  41. if (empty($url_title)) {
  42. echo 'Error setting URL title from '.$title;
  43. return;
  44. }
  45.  
  46. // check if we need to do anything. we won't bother making a link with
  47. // no title or url.
  48. if (empty($link_url) || empty($title)) return;
  49.  
  50. // post to textpattern
  51. $query = "INSERT INTO ".PFX."textpattern (
  52. Posted, AuthorID, LastMod, LastModID, Title, Body, Body_html,
  53. Annotate, AnnotateInvite, Status, textile_body, textile_excerpt, Section,
  54. Keywords, url_title, override_form, custom_1, custom_2, custom_3)
  55. VALUES
  56. ('$posted_date','$author', '$posted_date', '$author', '$title', '$body', '$body_html',
  57. '$annotate', '$annotate_invite', '$insert_with_status', $textile_body, $textile_excerpt,
  58. '$insert_into_section', '$keywords', '$url_title', '$override_form','$link_url', '$via', '$via_url')";
  59.  
  60. echo '<!--'.$query.'-->';
  61. safe_query($query);
  62.  
  63. // post to del.icio.us
  64. $posted_date = date('Y-m-dTH:i:sZ');
  65.  
  66. $client = new HttpClient('del.icio.us');
  67. $client->setAuthorization('user', 'password');
  68. $sucsess = $client->get('http://del.icio.us/api/posts/add',
  69. array('url' => $link_url,
  70. 'description' => $title,
  71. 'extended' => $body,
  72. 'tags' => $keywords,
  73. 'dt' => $posted_date));
  74.  
  75. ?>
  76.  
  77. Download this code: /static/code/4.txt

|  

Comments

  1. Hi ramanan,

    nice write-up. I, too, was thinking about using a custom_field for the referrer. The thing that turned me off, was that Textile always wraps the body in < p> tags. And I wanted to the via-link to appear at the end of the link-description, within the same paragraph.
    Other than that, the solution is even better. Maybe you are already thinking of the possibilites of making reports on who you referenced the most in July and other such “toys”. ;)

    > The challenge at this step was to
    > get them to display as lists.

    Wow, I am impressed. I stood before a similar, yet somewhat simpler problem (at least mine were not intermingled with regular articles), and I chose the easy way out, using not so nice markup (On my links-page each item is enclosed in it’s own < ul>). Nice work!

    btw: Sencer is actually my real name. :)

  2. I used CSS to display all the paragraphs that are generated for each list inline. I’m not sure if this is the best solution or not. One option would be to turn off textile for these links completely, which I may do at some point in time. Choices choices.

  3. Also, if anyone wants the code for any of this just ask.

  4. People have asked a few times now, so I’ve posted the code I use to make my link log. Also, I am using this funky new code display plugin I think is pretty cool. It’s based on something Dunstan at 1976design did.

  5. This is great code, Ramanan—thanks again. I would love to be able to assign categories to my links, so I have been tinkering with it a little, trying to get the linklog to treat categories nicely; as it is, the if_next/previous_section_different functions aren’t sensitive to category, so the linklog divs get munged up. So far, I haven’t had any luck, although I have managed to create some nice infinite loops trying to re-retrieve the neighboring entry. :)

  6. Pardons if I’m mistaken, but do you have the php you use to duplicate your links both within textpattern and at del.icio.us posted? I’d love to see that, I’ve been trying to get your kind of function into my linklog for a little and why reinvent the wheel?

  7. I’ve posted the code above. Hope it helps. You’ll need to make a front end to the script where you supply the various POST variables needed.

  8. Ok, thanks – my goal is to get it into textpattern so whenever I post an article to section x, it gets mapped to del.icio.us – but I’m expecting it to be way too much work.

    Thanks again!

  9. Oh, interesting idea. I don’t think it’ll be too much trouble. You will have to edit the file that does the actual saving of articles so that it checks what section the article it is saving is in, and if it is the appropriate section it will run that del.icio.us code. Oh, that code uses an external library. I don’t know if you noticed.

  10. How could I alter the code to not output the starting code if it’s the first post on the page?
    I’m doing the same sort of list, only not bothering with semantics and doing the links with divs. So i’m only using the if_next_section_different tag.

    ‘Cept, in our design, the first bar is a little out of place (see URL).

    Thanks in advance.
    -jb

  11. If you follow the changes outlined above it should do this. The article_num variable is what lets you know a particular post is the first (or last) post on a page.

  12. I got it, thanks.
    Another question: how did you do your XML feeds?

  13. What do you mean? What about them?

  14. How did you set them up to show links properly? With the ”[Link]” arrangement and all.

    TXP is great, but not-so-flexible in the XML sense.

  15. Modified the source code for the RSS and Atom files. I added this line of code: if ($Section == 'linklog') $Body .= ' [Link]';. It’s line 95 in atom.php and line 65 in rss.php.

  16. I’ve been re-visiting my linklog to add some technorati keyword functionality (via compooter’s plugin: mysql_num_rows(): supplied argument is not a valid MySQL result resource in /home/schussat/public_html/textpattern/lib/txplib_db.php on line 106), and I took a look at your code that posts to txp/del.icio.us. It seems to work fine (links are posted to both sites) but it does give me a warning:

    “Warning: mysql_num_rows(): supplied argument is not a valid MySQL result resource in on line 106”

    This seems to come from line 35, where you call safe_field and supply rsx_linklog as the name of $table. I can’t quite make sense of this. Is that particular to your installation, and should it be changed for others?

  17. Oh when I implemented this on my site I made a new table, rsx_linklog, which keeps track of the number of links I have posted so far. That’s why the URL-titles on my links are something like “link-100”, “link-101”, etc. It’s there for my URL scheme.

  18. Gotcha—neat. So, I should just replace “rsx_linklog” with “textpattern” to use my single table for txp entries?

  19. You should replace the chunk of code that generates the url title. I’ve done so in the example I have posted here, so the url_title is the stripSpaced() version of the title.

  20. Sorry for the comment bloat. That new code works great but has two tiny typos: $Title should be lowercase $title, and the line lacks a semicolon at the end.

    Thanks again, Ramanan. This is a great enhancement to txp.

  21. I’ll clean up the comments here in a bit. Thanks for the correction, i’ll fix it now.

  22. I have been hacking around fruitlessly trying to get this to work in 4.0.3

    I can’t seem to get the article_num variable to set and increment. The publish.php code is quite different to 4.0.1

    Any advise would be gratefully received.

  23. I’ll write up a revised version of this document tonight. Hopefully that will clear things up.

Don't be shy, you can comment too!

 
Some things to keep in mind: You can style comments using Textile. In particular, *text* will get turned into text and _text_ will get turned into text. You can post a link using the command "linktext":link, so something like "google":http://www.google.com will get turned in to google. I may erase off-topic comments, or edit poorly formatted comments; I do this very rarely.