lardbucket: blog

12/25/2016

Merry Christmas, Once Again

Filed under: General — Andy @ 12:00 am

It seems these days I’m basically only doing Christmas updates on my blog, which is probably a shame. Unfortunately, I don’t have anything all too public to update on. Work is going well, but is of course almost entirely NDA’d and so I can’t really say much about it, and what I can say isn’t worth a blog post.

More generally this year, however, it seems as though a number of people have been discouraged by political, economic, or other changes over the year. I can generally agree with some of them, as it seems as though a number of US (and potentially, world) policies are likely to change for the worse over the course of the next year. However, it’s also worth pointing out that a number of good things have happened (or become noticeable) this year:

* USB C became popular (though it was approved in 2014)
* Apple (semi-)successfully refused to provide assistance unlocking a targeted iPhone 5C (and, according to them, does not have the ability to provide such assistance on newer devices)
* The United States has begun to normalize its relationship with Cuba
* The Dow Jones Industrial Average has hit yet another all-time high

And of course there were many other “regular” good events. I’m sure I’m forgetting many major events, too.

Despite this, things could always be better. My plan for the coming year – and I’d encourage others to do similarly – is to find more areas where I can help make positive changes, and push to make those changes. Be they individuals or organizations who simply need a bit of support, or entirely new projects, the coming year promises to be one where many important things will be decided (as with every year, of course), and it’s important to remember that simply sitting on the side (or Twitter, or Facebook, or etc.) and wishing for things to be better is insufficient.

And with that, I’ll leave you with a link to the seasonal Darlene Love cover of A Marshmallow World.

Merry Christmas, everyone!
Andy Schmitz

No Comments »

No comments yet.

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

12/25/2015

Merry Christmas, 2015

Filed under: General — Andy @ 12:00 am

And so we’re about to round out 2015. I haven’t done much public of note (other than what’s already documented on this blog), but hopefully everyone has had a productive and enjoyable year. Here’s to a 2016 that’s even better.

8 Comments »

  1. Dear Andy Schmitz, you have a pirated copy of my book “Business Ethics Workshop” up on your website, please take it down, thanks you, James Brusseau

    Webpage: 2012books.lardbucket.org/pdfs/business-ethics.pdf

    Comment by James Brusseau — 1/15/2016 @ 5:04 pm

  2. James Brusseau: That’s not really a comment on this post, but it’s also quite an inflammatory claim, particularly given that it is false.

    I assume you found my blog via the second page of that PDF, which notes that I am distributing it, as well as giving links to the project’s main page and attribution page, both of which explain in detail exactly how and why the books are legally hosted on my website.

    To briefly summarize because you’ve brought that discussion here: your book, along with many others, was available under a Creative Commons license (by-nc-sa 3.0 unported), directly from the publisher as of December 29, 2012. That license, which is non-revokable, allows anyone who obtains a copy of the book to redistribute it under the same terms, as long as it is not used commercially. That is exactly what I am doing in the PDF you’ve linked to, as well as in the browseable version at http://2012books.lardbucket.org/books/business-ethics/.

    Since the time I downloaded copies of those books, the publisher has removed their free access to the books. However, I can still redistribute the books according to the terms of the license they had when I downloaded them, and I am doing so. (The Creative Commons license gave the publisher a number of additional rights, such as asking me to remove the attribution statements generally required by that license, and I have done so per their request, which is also why I’m not naming them here.)

    I do find your comment interesting in light of section 1.1 of the book in question, and its discussion of the ethics (or lack thereof) of “kickback textbooks”, particularly given that your two upcoming Pace University Spring 2016 sections of PHI 121 list a digital copy of your own book as a “required package”. Perhaps your students will find it an amusing coincidence as well.

    At any rate, it is clear that at the time the book was downloaded, it was licensed in a way that allows me to redistribute the book in perpetuity, and that the licensing was intentional (indeed, that fact was widely used as a selling point by the publisher, both to encourage teachers to adopt the textbooks, and to encourage authors to publish with them). I certainly appreciate you choosing to publish your book in such a way, as it has made the content available to tens of thousands of additional learners.

    I would appreciate a retraction of your claim that I am in some way hosting a pirated copy of your textbook, as it is clear that this is not the case.

    Thank you,
    Andy Schmitz

    Comment by Andy — 1/15/2016 @ 6:31 pm

  3. Mr. Brusseau’s comment is not false. You have published my book under the wrong title “A Primer on Politics.” It wasn’t even published until 2013, so to say it was available by license in 2012 is bullshit. You need to take my book down. I have contacted my publisher; my attorney will follow.

    Comment by T.M. Sell — 3/23/2016 @ 9:28 pm

  4. Mr. Sell: A couple of things are worth making note of here. The above blog post has no context on the books in question, but it’s the venue you’ve chosen for some reason, so I’ll have to give a bit of background.

    The books published on 2012books.lardbucket.org are books that were downloaded in 2012, with no exceptions. They were downloaded from a publisher who made all of their books available under a Creative Commons license through the end of 2012. I then made those books available pursuant to the terms of that Creative Commons license, which allows redistribution under a few conditions (to which I have scrupulously adhered). Notably, the Creative Commons license is non-revokable, and the fact that the publisher no longer makes all of their books available under the license is irrelevant.

    The reason I am not naming the publisher here is that they later contacted me and asked me to remove the authors’ names as well as their own name from the attribution of all of the books. They are allowed to make this request under the Creative Commons license they chose, and I am required to adhere to it, so I have. They further requested that I change the name of all of the books to not match the names they were using to promote the books, and again I have complied with their request. (This is the reason that your book is published “under the wrong title” – were it not for your publisher asking me to change it, I would have used the original title and attributed it to you.)

    As for the claim that your book “wasn’t even published until 2013,” I can assure you that the copy of the book that I am hosting was indeed available in 2012. It was listed at http://catalog.[publisher's domain name].com/bookhub/reader/5913 at the time I requested it, and the publisher’s server helpfully responded including a header with the date: “Date: Sat, 29 Dec 2012 05:15:42 GMT”. This was firmly inside 2012 by any definition and time zone. At that time, I verified that the book listed a Creative Commons license. The actual content of the book (as it was spread across several pages) was downloaded between “Sat, 29 Dec 2012 06:41:34 GMT” and “Sat, 29 Dec 2012 06:41:39 GMT”, as reported by your publisher’s server, just over an hour later. Finally, after I finished downloading everything – and before the end of 2012, I went back once again, and verified again that the Creative Commons license was indeed still present on each book, including yours. I have thorough documentation of all of this, including additional detail and full responses from your publisher’s server.

    What may explain the apparent discrepancy is that the book I have listed on my website has disclaimers on virtually every page, added by you or your publisher, indicating, “PLEASE NOTE: This book is currently in draft form; material is not final.” So, while the final copy of the book may have been published in 2013, the draft copy was available online, and was licensed under the Creative Commons by-nc-sa 3.0 license, toward the end of 2012, when I downloaded it. This draft copy, available under the Creative Commons license in question, is the one I am hosting.

    As was the case when Mr. Brusseau made similarly incorrect accusations, I would appreciate an apology. It is clear that I am only hosting copies of books which were available under a Creative Commons license at the time I retrieved them.

    Comment by Andy — 3/24/2016 @ 10:01 pm

  5. Hello!
    Please can you email me the the author, year of publish, publisher of the book Primer on Communication studies, the pdf is very helpful for an assignment, however at the moment I am unable to use it as a reference because I am required to Harvard reference it.

    Many thanks!

    Comment by Student Z — 4/16/2016 @ 5:35 am

  6. Student Z: Unfortunately, the publisher of that book has asked me to remove all attribution from that book, so I’m unable to properly attribute it to the author. You may be able to find the original by copying a sentence or so and doing a search for it, which may return the original publisher’s copy. (I don’t actually know if that will work in this case, as I haven’t tried it, but it frequently does for other books when I’m trying to find their sources.)

    Comment by Andy — 4/17/2016 @ 10:23 am

  7. Andy!

    We went to High School together, and it’s kind of goofy I came across your blog today when looking up a map of Southwest Asia. Crazy how that works out. Thanks for providing so many resources on here.

    Had a question about something kind of…out there, but I figured you know computer stuff really well and could assist me. I need to get a full text RSS feed for my blog in order for another website to post my blog posts on their site (I can go into more detail if needed!), but everything I’ve offered them (www.crunchtimetwolves.com/feed and utilizing another full text feed website) doesn’t seem to provide the full text they need….any ideas?

    Sorry for dump this on you in your comment section like this! Hope things have been going outstandingly well for you after all this time.

    Comment by Jonathan Slusar — 5/2/2016 @ 4:06 pm

  8. Jonathan Slusar: Jon! Crazy indeed. To show the full content of WordPress posts in the feed, you should be able to log in to WordPress, go to the Settings page, the Reading tab, and under “For each article in a feed, show”, pick “Full text”. After that, your feed should indeed start showing the full text for each post, rather than a snippet. Good luck! (Sometimes themes might override that setting, but they’re not really supposed to.)

    Comment by Andy — 5/4/2016 @ 7:34 am

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

5/31/2015

Sunsets

Filed under: General — Andy @ 8:27 pm

sunset

If you’re a U.S. citizen under 27 years old, the sunsetting of the USAPATRIOT Act’s Section 215 provision at midnight tonight will mark the first time since before you were a teenager that the U.S. government will not have legal cover to collect data on every phone call you make. (Barring yet more secret interpretations of existing laws.)

Reasonable people can have reasonable debates about what surveillance should be allowed. For example, it’s reasonable to allow police to follow a murder suspect. However, these debates must be had, must be had in public, and must include the public. The USAPATRIOT Act had none of these qualities, and it is high time that some of its sections have now “sunsetted”.

While several portions of the act will likely be reauthorized in the coming days, potentially with some reforms, it is important to not simply accept minor changes as the only possibility. We can – and should – continue to protect privacy as an important civil liberty.

No Comments »

No comments yet.

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

4/20/2015

PlaidCTF 2015: radhos Writeup

Filed under: Ego, Hacks, Programming — Andy @ 6:44 pm

Along with several coworkers, I participated a bit in the PlaidCTF 2015 competition this past weekend. The competition is geared mainly toward binary exploitation, with reverse engineering a secondary priority. There are also a number of challenges which focus on cryptography, forensics, and other security-related issues. Unfortunately, my skills lie more toward the latter category than the former, so I wasn’t as broadly useful as I might have otherwise liked to be. Nonetheless, I was able to contribute solutions to three problems: “RE GEX” (a large regular expression, where it was necessary to find a string that didn’t match the regular expression), “PNG Uncorrupt” (a PNG file with several missing “\x0d” bytes before “\x0a” bytes), and “radhos” (a Python hash collision challenge). I’ve seen good writeups online for RE GEX and PNG Uncorrupt, but haven’t yet seen any good ones for radhos, so I’m contributing my solution here.

You can download the radhos challenge here (or here if the first link has disappeared). If running it locally, replace line 24 with “ return '127.0.0.1'” to keep the server from having errors.

radhos is a simple HTTP server that emulates a small hashtable-based key-value store. Each value is stored using the hash of its key, where the hash is the lower 32 bits of Python’s hash() function. (This appears to be intended to give consistent behavior between 32-bit and 64-bit versions of Python.) The level’s flag is stored in the hashtable’s location for the key “you_want_it_LOLOLOL?”, but the server will reject requests that ask for that key by name. Therefore, the remaining solution is to find something that will generate a collision in hash(key). To encourage players to solve the challenge without simply guessing random keys, the server restarts (and gets a random hash() seed) every seven minutes.

Hash Collisions

Using HTTP requests, you can get the value for a key you select, or alternatively, you can set the value for a key. Setting the value for a key will also return the old value in that location in the hashtable, if any, as well as the value of hash(key). It’s this information leak that allows us to break Python’s randomized hash function, generate a new key that has the same hash value as “you_want_it_LOLOLOL?”, and retrieve the flag.

I initially thought that this would be an instance of the bug described at https://bugs.python.org/issue14621, but that doesn’t appear to be the case. I spent far too long looking at that issue, and trying to replicate similar situations that might allow a collision in this case. However, the issue does have an attachment, hash.py, that gave Python code for the hash algorithm, which was somewhat useful for analysis. The (pre-3.4) Python hash() function used two secret values to obscure the hash value: a “prefix” that affects the whole hash, and a “suffix” that is simply XOR’d in afterward to obscure the result and prevent attackers simply guessing the prefix. If an attacker is able to determine the “prefix”, they can generate hash collisions on their own. (Having the suffix is unnecessary if collisions are the only goal, as it is simply XOR’d on at the end.)

Here’s a bit of code for generating the hash:

HASHMASK = 0xffffffff

def hashbytes32(data):
    if len(data) == 0:
        return 0
    x = prefix
    x ^= ord(data[0]) << 7
    for byte in data:
        x = (1000003 * x) ^ ord(byte)
        x &= HASHMASK
    x ^= len(data)
    if x == -1:
        x = -2
    return x

Finding the prefix

Luckily, there was a talk on hash collisions (Hash-flooding DoS reloaded: attacks and defenses) that referenced a tool that could be used to determine the hash secret in Python. I was able to find the tool at https://131002.net/siphash/poc.py, and verified that it worked.

Unfortunately, it worked based on hashes for “\x00” and “\x00\x00” (one and two null bytes), which we couldn’t submit to the radhos server (as far as I know). A small amount of math allows us to change the target used in the algorithm, however. Let’s say that we have h1 = hash("A") and h2 = hash("B"). Then we know:

h1 = ((prefix ^ (65 << 7)) * 1000003) ^ 65 ^ 1 ^ suffix
h2 = ((prefix ^ (66 << 7)) * 1000003) ^ 66 ^ 1 ^ suffix

Let’s XOR the two, and note that the 1 and suffix terms cancel out, as they are on both sides. Then we have:

h1 ^ h2 = ((prefix ^ (65 << 7)) * 1000003) ^ 65 ^ ((prefix ^ (66 << 7)) * 1000003) ^ 66

The tool I mentioned above uses this fact, and guesses the value of prefix one bit at a time, narrowing down possibilities as it goes. It then performs a verification step, testing those possibilities with a known hash of a third value. I was able to verify that I received the correct prefix value by testing locally, so I was satisfied with that part of the code.

As an aside, if the bit-by-bit guessing didn’t work to determine the prefix, it would be possible to use the h1 ^ h2 math above and several additional hashes to brute-force the prefix, but that would likely take a while. Distributing the problem among multiple cores (or computers) might get a result before the server restarted, however, so it might have been worthwhile if the proof of concept code were not available. (Note that brute-forcing the suffix is unnecessary in this case, as XOR’ing the hashes removes it from consideration.)

Finding an actual collision

Simply having the prefix value is insufficient, as we need to actually generate something that will collide. The obvious solution is a brute-force attack. This is easy to implement, and can be parallelized to get a result before the server restarts. Unfortunately, it’s also fairly slow in general, when another relatively simple solution is available.

Because the hash code is relatively simple, we can perform what is known as a “meet in the middle” attack. In this case, we start with the result we want (we can already calculate the hash for “you_want_it_LOLOLOL?”, because we have prefix and suffix
mask = 0xffffffff
def hashback(instr, target):
val = target
for byte in instr[::-1]:
val = ((val ^ ord(byte)) * 2021759595) & mask
return val

This works because 1000003 * 2021759595 = 1 mod 232, so multiplying by 2021759595 is equivalent to dividing by 1000003. (Note that in this function, for simplicity, we have already removed the suffix and the size of the string to be hashed.)

So we’ve worked backward and determined what the hash state should be for a given suffix. How does this help? Well, we can do the same thing for many suffixes, and have many possible targets for our brute-forcing. The most efficient way to do this is to try 216 suffixes. This turns the brute-forcing from an expected 232 operations with 1 unit of storage into 2*216 operations with 216 storage. RAM is relatively plentiful for this task, so it’s a good trade-off. I’ll let the code below speak for itself, but feel free to leave a comment if you have a question.

Putting it all together

The code I ended up using is below, after some minor cleaning up:
(more…)

No Comments »

No comments yet.

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

11/30/2014

2012 Book Archive: Now With PDFs

Filed under: General — Andy @ 10:58 am

Almost two years ago, I launched (what became) my 2012 book archive. There’s a bit of background on that project on that page. While there have been a few minor developments since then, none of them have been noteworthy enough to document here. Recently, however, I decided that I would try to make PDF copies of the books. I wanted these to be good-quality PDFs, as although a number of PDFs of the books had circulated from other groups in the past, they weren’t particularly visually appealing: they were essentially what you would get by quickly printing each of the HTML files from the publisher.

My guess was that a simple HTML file with all of the book content could be combined with a quick print CSS style and get good-quality output without too much effort. I figured that with a decent number of iterations on the book CSS and test runs, the project might take a week or so. Over two months later, I’m finally set to release the PDFs. I’ve made a few notes of the things that I did along the way, in case anybody finds them useful in the future.

If you just want the PDFs, you can visit the 2012 book archive and download them there. There are a few ways to download each book: you can download a whole-book PDF file, or a PDF for each chapter. Both are accessible from a book’s table of contents, and the whole-book PDFs can be downloaded from the archive’s book list as well.

Getting a PDF

It’s worth noting that the content I have from the books is already in an HTML format, not necessarily the raw input to a book creator. I decided early on to try to work with HTML and CSS, because although the HTML I have is structured and could probably be parsed into a different format, doing that correctly for over a hundred books would be iffy. With that said, the HTML was well-structured, and lent itself to easy use of CSS to style specific things.

If you’ve been following CSS development, there are a lot of features in CSS that are supposed to allow styling of printed content. Unfortunately, support for some of these in browsers is spotty, and they don’t necessarily provide all of the features that you’d like when styling a book. (In particular, footnotes, page-relative positioning, page numbering, and PDF bookmarks are all difficult. There are likely other features I’ve forgotten about as well.)

At first, I was hoping to be able to use something like wkhtmltopdf, which wraps WebKit and converts an input file to a PDF. This gave me a number of problems, and didn’t seem to support the concept of pages as natively as would be desirable. (It’s still impressive that they managed to get the project to work as well as it does, but it doesn’t produce nice-looking books yet.) After that, I decided that perhaps Firefox’s support for printing would work for me: it works pretty well with many of the CSS printing features, and I can probably script Firefox to output PDFs if I want. Unfortunately, I again ran in to bugs with the rendering. I don’t recall the details at the moment, but I believe that content had a tendency to not wrap between pages in the right spots. Either way, this sent me in search of a high-quality PDF rendering solution.

If you look online for advice about generating PDFs from HTML, you will inevitably run upon many people suggesting PrinceXML (or, as it seems to be rebranding itself, Prince). They’re probably right. It is a commercial piece of software in a case where I had hoped to use free software, but it is still the best solution I have found by far, both in ease of use and functionality.

Princely Things

Prince itself is not cheap. A personal license is $495 at the time of writing, and even that may not cover what I intend to do in terms of converting books automatically. (To be clear, it might be covered, but only just barely if it is. I haven’t asked, for reasons I’ll explain shortly) If you are doing anything serious with Prince, you’re probably looking at a $3800 license per server generating PDFs, or a $1900 one if you’re only doing academic things. Upgrades are available for an additional annual cost. To be clear, if you’re generating revenue from the PDFs (or even just saving yourself loads of time), Prince is almost certainly worth every penny, but it’s prohibitive for side projects.

For non-commercial projects, Prince offers a free version with the requirement that you allow it to add a logo and link to the corner of your document’s first page, link to their website wherever you have Prince PDFs for download, and link to their website on a sponsors/partners page. This is mostly unintrusive (although a tad confusing at first: I’ve considered trying to style in a little “Made with” above the logo to explain why it’s there), and very nice of Prince to allow. (To get the “Non-commercial” license, just download the software: you don’t need a special license key or anything.)

In fact, I had a question about their licensing (“The books are licensed under a Creative Commons license that doesn’t allow me to add restrictions to them, so is it required for people who receive the PDFs from me to keep the Prince logo on them? If so, I can’t use the noncommercial license.”), emailed them, and got an email back quite quickly from Håkon Wium Lie, Prince’s Director (not to mention CTO at Opera and founding member of the Pirate Party of Norway). He’s definitely on top of things, and was quite happy to help. (The answer is no, other people can do whatever they want to the PDFs. In my case, they’re still subject to the Creative Commons license they always were, but that’s not because of Prince.) Later, I had a question about how to get something to render correctly (a somewhat minor, obscure layout bug), and quickly received a comment from Mike Day, the CEO, noting that they were looking into the issue. When I followed up, the bug hadn’t yet been fixed (it undoubtedly has tricky interactions with their page layout code), but I quickly received an alternative suggestion complete with example code. Definitely a pleasant experience all the way around.

If you’re looking for a cheaper option to start with, you’ll probably run into DocRaptor as well. DocRaptor started out as Prince-as-a-service, providing an API to allow people to generate PDFs using Prince. It now appears to support Excel files, although I haven’t looked in to those features. For many people, the benefits of being able to rely on DocRaptor to scale up as your workloads do (they claim “thousands of documents a second”) and the lower initial costs are probably a great benefit. They also provide well-supported libraries for a number of languages, where Prince usage is largely done by command line (although Prince has a PHP API as well). Overall, DocRaptor almost certainly provides benefits for many people. However, their plans aren’t super cheap either, and they’re targeted at recurring use, not one-shot uses like mine. I generated over 2500 PDFs in my final output (one per book, plus one per chapter), which would probably have cost me $149 in a month, assuming I didn’t want to tweak them later. Still far cheaper than the cheapest Prince license, but pricey for a personal side project like mine.

DocRaptor does have a 7-day free trial, which probably would have allowed me to generate whatever I wanted during that time, but that’s not exactly ideal, either. (Nor do I mind paying something for the service, but over a hundred dollars a shot is high for my purposes.) I emailed the DocRaptor folks about a pay-as-you-go plan (so I wasn’t paying monthly fees when I wasn’t using the service), because I had found references to such a plan elsewhere. I got a very nice response from Matt Gordon, the “lead vocalist” for the group running DocRaptor. Unfortunately, they no longer offer that plan, because they found that disproportionately more of their support costs (and they do provide good support) were going to users who didn’t spend much on the service anyway. We had a nice conversation about the possibility of plans that might support alternative uses such as mine, but it doesn’t sound like there’s anything planned in the immediate future. (I can’t blame them, as they need to make money and do what makes sense for their business to continue existing.) They did make a very nice offer (I won’t disclose the details) that I turned down for unrelated reasons, but they’re definitely nice folks too.

My conclusion is that you pretty much can’t go wrong with Prince or DocRaptor. Both have very nice and responsive folks behind them, and seem to be quite well done.

Tables of Contents and Bookmarks

One of the things relatively unique to printed books is cross-references with page numbers. Most of the book content doesn’t include these. This is primarily because any existing cross references are links to a specific section, and I didn’t think it necessary to include a page number along with the section number. However, the table of contents for the book definitely benefits from page numbers. Pulling a table of contents together in Prince is relatively easy. It could possibly be done automatically with JavaScript, but I chose to create taables of contents in a Ruby preprocessor as I was assembling whole-book files anyway. Prince makes it easy to include page numbers for links to given anchors, so I only needed to pull out the anchor for each section. (Luckily for me, I already had the anchors in a database.)

Secondly, I wanted to make sure that chapters and sections were listed in the PDF list of bookmarks. This list is sometimes useful when navigating a book in a PDF viewer, although some viewers don’t show it. Prince again makes this quite easy, simply requiring a CSS annotation for the items you wish to be bookmark headings. (In fact, by default it uses h1-h6 tags, but I disabled that default because it picked up way too many bookmarks.)

Optimization

In creating the full-book files, I noticed that some books created particularly large files. In general, this appeared to be because they embedded the full source images, rather than resampling them. While an option to resample the images inside Prince would be great, it doesn’t exist at this time. Some of the source images were quite large, and clearly intended to be printed at >= 300 dpi, while most users of the PDFs wouldn’t benefit from such images. My first attempt at reducing file size was to use Ghostscript to resample the images. Ghostscript has some features that work similarly to the now-unavailable Acrobat Distiller, and seemed likely to do the job. Unfortunately, after getting Ghostscript working (Ubuntu 14.04’s version appears to crash on larger documents, but 14.10’s works), I found that it removed page numbering information and bookmarks. The next step was to try to export this metadata using PDFtk before using Ghostscript, and then import it again afterward. Unfortunately, while PDFtk will output page numbering details, it won’t import them into a PDF, and there doesn’t appear to be any easily-available way to do so.

So, I temporarily abandoned the option to resample the images using Ghostscript. (It also may or may not have been worth it in the first place: some Ghostscript-generated files were larger than the Prince originals, so I had to handle both cases.) It may be worth patching Ghostscript in the future to keep the metadata around, but that seems likely to be quite involved. In many cases, you may get some benefit out of using Ghostscript with appropriate options (“gs -q -sDEVICE=pdfwrite -dPDFSETTINGS=/printer -dBATCH -dNOPAUSE -sOutputFile=[outfile] [inputfile]” seems to work well besides the additional metadata), but it was unfortunately unsuitable for my purposes at this time.

Chapter Files

Following up on the “whole book PDFs can be pretty big” issue, and after trying to open some such books and experiencing slow loading times, I decided that it may be appropriate to create one PDF per chapter as well as the whole-book PDF. My first pass at creating these PDFs was to use PDFtk to pull out just the pages from a given chapter. This posed a few problems: first, I had to figure out which pages belonged to which chapter. Luckily, the bookmarks inserted by Prince, combined with PDFtk’s metadata output, gave me the starting page for each chapter (although for a few minor reasons, this link was a bit iffy: the generated bookmark title did not always match the section name I had in my database), and I could assume that a chapter ended just before the next one began. Unfortunately, this ran into the same problem I had before: I would lose the page numbering and bookmarks. (Not to mention the fact that I would need to separately render a new first page to describe the licensing and get the Prince logo back on the first page.) Finally, I decided to simply depend on Prince once again.

I got Prince to log the page number and ID for each chapter heading by using a ::before psuedo-class with a content property of “prince-script(log, counter(page), attr(id))”, and a small “log” function in the JavaScript on each page. This allowed me to use the IDs to match up with my database, and easily identify where each chapter started. Because I already had whole-chapter HTML files, I could then use those HTML files to render the chapter in Prince, and everything would still be in sync, without having to try to render and merge together separate front pages for each chapter. (I still needed to get the page numbers to Prince for rendering purposes, but for this, I simply placed the page number in a CSS block in the HTML file.)

This solution appears to have worked surprisingly well, with the page numbers matching up where expected. Because the files were rendered separately, there is the possibility of some unforeseen issue (I certainly didn’t inspect the thousands of files by hand), but it seems unlikely.

Math

Finally, when reviewing one of the math textbooks in the collection, I noticed that Prince’s MathML rendering wasn’t particularly great. It is definitely better than nothing, but the rendering quality did leave something to be desired. Unfortunately, the most common web-based solution here, MathJax, doesn’t work very well with Prince. (This is a noted todo item on Prince’s release notes, but it’s not available yet.) After stumbling through a number of other options to try, I ended up using PhantomJS together with MathJax to prerender the math to MathJax’s “HTML-CSS” output (the SVG output didn’t look very good and produced a very large PDF file after the required fixes to make Prince display the SVG output). I forced MathJax to use the STIX fonts (which I installed on my computer), and after the math was rendered, I output the document’s HTML form again (after removing the MathJax wrapper divs). This produced files with reasonably good-looking math, the way they were intended to look. The prerendering code hasn’t been published yet because I haven’t taken the time to clean it up, but if someone is interested, I can definitely post it.

Prerendering with MathJax is a step that seems to have very poor asymptotic time complexity. I haven’t formally benchmarked it, but a chapter’s sections took about two minutes to prerender in total, while the whole chapter itself took roughly twelve minutes to prerender. The whole book took roughly four days to prerender. It’s not clear why this occurred, but the prerendering did eventually succeed. It’s also not clear if this is a bug in MathJax, or simply some inefficiency in PhantomJS, so I have yet to report it as a bug to either project (and may never report it – it’s unlikely to come up in common use).

Fin

So, to summarize, getting PDFs of a quality I’m comfortable with took quite a bit of effort. In the end, Prince does most of the work, and I rarely had problems with Prince itself. I think it was worth the effort, at least for a personal learning experience. Hopefully the books will be useful to other people as well. Once again, they’re all available at http://2012books.lardbucket.org. Please feel free to copy or redistribute them as you see fit, pursuant to the terms of the associated Creative Commons by-nc-sa license.

Andy Schmitz

P.S. If you’re interested in any of the print-specific (or Prince-specific) things I did to make the books look decent when printed, it’s all left in the book’s CSS file toward the bottom, under the “prince” @media type. Feel free to reuse any of that styling for any purpose you see fit, in any situation. I do not believe it is covered under the Creative Commons license: you may consider it to be public domain.

17 Comments »

  1. Hi there Andy. Can you advise on who to contact for permission to use material from one of the books in the 2012books archive for commercial use? Thanks!

    Comment by Ilka — 12/11/2014 @ 7:37 am

  2. Ilka: Unfortunately, I don’t have the ability to license the books for commercial use, as I am not the original author (or publisher). The publisher has asked to remain anonymous, but I will contact you privately to see if it is okay to pass along your information to them and have them contact you.

    Comment by Andy — 12/11/2014 @ 9:45 pm

  3. Dear Andy, do you know how to access business cases videos “How Would You Handle This?” from Beginning Management of Human Resources book? The videos ask for password (for ex. p.30). Thank you.

    Comment by Tetiana — 1/8/2015 @ 8:18 am

  4. Tetiana: Unfortunately, it looks like those Wistia videos have been removed. For the moment, they’re inaccessible. I do have copies of those videos that I saved when I downloaded the books, so I’ll have to see about getting those put up. Unfortunately, getting videos to display online isn’t particularly straightforward, so it will likely take quite a while for me to get them all together.

    Comment by Andy — 2/15/2015 @ 12:47 pm

  5. Hi Andy,
    Thanks for your work preserving the Creative Commons book archive. I had bought individual chapters from Flatworld for my students under their old business model, but they no longer offer that fee structure and I didn’t want to buy their entire books when I only used a couple chapters, so I stopped assigning the chapters. It is nice to be find the books for use as a reference again.
    I’m thinking about trying to move more of my reading material over to a Nook or Kobo e-reader. My ancient Kindle just broke, so I need to do something different anyhow. I’m wondering if you have any plans for posting these in the EPUB format or if you have any advice about reading them on a EPUB reader?
    Thanks,
    -Jonathan

    Comment by Jonathan Andreas — 4/2/2015 @ 11:24 am

  6. Jonathan Thanks! I don’t have any near-term plans for posting the books in an EPUB format, mostly because of my currently oversubscribed free time. It’s a reasonable idea, and shouldn’t really be all that difficult: EPUB is basically an HTML document anyway. Unfortunately, if I do end up making EPUBs out of the books, I would probably want to spend a bunch of time making sure I got everything right. (Obviously some things, like embedded YouTube videos, wouldn’t transfer well, but most things should be fine.) I’ll put it on my list, at any rate. Sorry about that!

    Comment by Andy — 4/2/2015 @ 4:50 pm

  7. What’s the difference between your two files for “Principles of general chemistry”? What does “1.0M” have that “1.0” doesn’t?

    Comment by Jay — 4/13/2015 @ 1:10 pm

  8. Jay: That’s a good question. At a quick glance, I wasn’t able to see any differences in the table of contents, although it is possible that some of the content is different. I presented them as separate books because they were separate books in the original source of the Creative Commons books, but I wasn’t able to determine why they were separate. Sorry I couldn’t be of more help!

    Comment by Andy — 4/20/2015 @ 6:57 pm

  9. It’s my honor to write and commend you on your effort,even though i could get access to you publication.

    Comment by IBRAHIM — 4/22/2015 @ 10:06 am

  10. Dear owner of 2012books.lardbucket.org/books Website

    Many thanks for posting books on your website! A student in my class has brought it to my attention that the atomic mass of Ag (silver) in the Periodic Table in section 2.7 in the book “Introduction to Chemistry: General, Organic, and Biological” (v. 1.0) is incorrect. The atomic mass should be 107.868, but it is listed as 196.56655. If this can be corrected, it would be great because if would help students who search for basic information from the web.

    Thank you very much for paying attention to this issue.

    Best regards,

    Youxue Zhang
    Professor, Univ. Michigan

    Comment by Youxue Zhang — 9/20/2015 @ 7:38 pm

  11. A colleague of mine and I wanted to use a diagram that I found in one of the books from Kurt Lewin in a book we are writing. How do I get permission to use that diagram?
    Thanks!

    Comment by Anil Saxena — 9/24/2015 @ 11:12 am

  12. IBRAHIM: Thank you! Are you indicating that you can’t view them? If so, can you give a bit more information on what’s not working? Thanks,

    Youxue Zhang: Thanks for the correction! I’ll try to update it soon. Unfortunately, I’ll have to figure out a way to do so, as I’ve thus far tried to keep everything very similar to the books as published, so it may take a while.

    Anil Saxena: Thanks for asking. Unfortunately, I don’t have any rights to sublicense the books. If you can work with the Creative Commons license listed on each page, then you don’t need any extra permission. If you need something beyond what’s allowed by that license, you will need to reach out to the current rightsholders. They have wanted to avoid me directly naming them, so I will contact them and see if I can forward along your request.

    Comment by Andy — 10/26/2015 @ 5:01 pm

  13. Hello,
    Thank you for the work you are doing to make textbooks accessible to all students. I am a community college instructor working to create a free textbook for students. My goal is to combine text from two of the Lardbuck textbooks to create a single composition and analysis book my students can use. My question is about attribution. Do you have any guidelines regarding citing our use of full and partial chapters? For example, should we include source information at the beginning of each chapter, before or after each section borrowed, in the preface, footnotes, in-text citations? I look forward to hearing from you.

    Comment by Tammy — 3/5/2016 @ 10:53 am

  14. Hello, I’m writing to request information about attributing the cc-sa-by textbooks. I’m an instructor at a community college and we are interested in utilizing chapters of several of the textbooks in our attempt to create a first year composition textbook for our students. I am hoping to learn how you expect users to attribute sections of the text (chapter introduction, foot notes, in-text citations, preface). Thank you.

    Comment by Tammy — 3/7/2016 @ 1:18 pm

  15. Hi Andy,

    The .PDF link for Advertising Campaigns: Start To Finish seems to be broken. I click on the .PDF link but I just get a black screen. Any advice would be appreciated, thank you.

    Comment by Steve G — 3/9/2016 @ 12:02 pm

  16. I have read through the licensing but still have a question. There are chapters of the book that I do not want to use, and I would like to add some of my own material. Can I delete or add material to the book? If so, is there anything that I need to do to modify the book? Thank you for your work on these books!

    Comment by Lynne — 5/31/2016 @ 8:15 am

  17. Sorry for not getting back to everyone sooner!

    Tammy: Unfortunately, I can’t really offer great citation recommendations, other than “cite them as you would anything else”. The Creative Commons organization has a set of best practices for attribution that may cover what you need. Because the original publisher has asked for the authors’ names and the publisher’s name to be removed from the attribution, you may be best off citing “an unnamed author”. You are welcome to link to my copies of the books as well, as the original publisher has asked that the copies not be linked to them. (I am also not a lawyer, and if you’re concerned about the legal aspects of citing a Creative Commons work, you may wish to talk to a lawyer.) Don’t forget that you can’t use the CC by-nc-sa license commercially. If you’d like me to get you in touch with the original publisher to talk to them about a different license, let me know.

    Steve G: The full PDF file for Advertising Campaigns: Start to Finish seems to load for me. It’s somewhat large, so it takes a bit to load, but it does successfully load. Can you try again? If it doesn’t work in your browser, try right-clicking the link, saving it, and opening the resulting file instead.

    Lynne: Yep! The Creative Commons license allows you to modify the book and reuse it, as long as you don’t do so for commercial purposes. To do so, just make sure you attribute the chapters that came from other authors to them. (In this case, that may be to “an unnamed author”: see my response to Tammy just above.) You’ll need to make sure you follow the license that’s linked to from the top of each chapter, but that should be relatively simple. Unfortunately, if you want to sell access to the books, you would need to get a license from the original publisher, as I don’t have the ability to give that permission. If you want to talk with them, let me know, and I can put you in touch.

    Comment by Andy — 6/17/2016 @ 4:56 pm

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

3/3/2013

Calcsy – See your calculator on your computer

Filed under: Education, Math, Programming, School, Technology — Andy @ 9:17 pm

Calcsy LogoCalcsy is a tool you can download and use on your computer to show (and save) the screen of your TI-84+ or TI-89 Titanium calculator.

It’s useful for projecting the screen large enough for other people to see, or for taking a screenshot to use in instructions. For example, I’ve used it in teaching how to graph functions on a calculator, and I suspect it will be similarly useful to other people.

I first wrote Calcsy almost two years ago, and have been (very) slowly making it better since then. It’s now to a state where I think it’s reasonable to release. At the moment, it’s only available for Mac OS X, although it should be possible to port to Windows if there’s enough demand. Hopefully it’s pretty self-explanatory, but feel free to let me know if you have any questions.

All you need is a TI-84+ (or TI-84+ Silver Edition) or TI-89 Titanium and a USB cable to plug it in to your computer. The program is free, and you don’t need any special software on your calculator. (You also don’t need one of the special “Presentation Link” adapters – your computer and a normal USB cable works just fine.)

Other details of note: I suspect it won’t work with the very new TI-84+ Color calculators, but I’m happy to try to make it work if someone wants to send me one. Also, the logo was made by David Felice, so thanks are due to him.

Anyway, it’s free, go check it out. Let me know if you have any questions/comments/problems/etc.

Andy Schmitz

3 Comments »

  1. Hello Andy:

    The university where I work has asked me to design some materials for an ESL course. I have seen useful material on your 2012books section, and I would like to use some of the texts you have posted there . Considering the fact that the University would somehow charge a fee for students wishing to take the course, would I still be able to some of that material?

    Juan

    Comment by Juan Arcila — 6/20/2014 @ 11:37 am

  2. Hi Andy,

    This comment is not in response to this blog post, but I wanted to contact you and did not know how else to.

    I am a chemistry professor and I have come across a periodic table published here: http://2012books.lardbucket.org/books/introduction-to-chemistry-general-organic-and-biological/s05-07-the-periodic-table.html

    It says you have something to do with this under Creative Commons. The reason I am bringing this up is because there are multiple typos and misinformation in this periodic table. It is one of the top search results on Google when periodic table is searched. As a result, every semester I have students who end up using it without realizing it is wrong, and they get wrong answers and miss points because of it. Is there any way you can take it down?

    Comment by Kay Dutz — 11/13/2014 @ 1:21 pm

  3. Sorry about taking roughly forever to respond to your comment on my blog: a deluge of spam had left my comments nigh-unusable. Luckily, I’ve gotten that mostly cleared out (thanks, Akismet!), and am now getting back to people.

    Juan Arcila: I am not a lawyer, and this is not legal advice, but my understanding of the noncommercial restriction on the Creative Commons license is that it is fine to charge for things around the books, but that you can’t charge for access to the books. You are definitely free to talk about them / assign readings from them/etc. in your class, and point to the copies on my website. I think the license means you can’t restrict access to copies you might make or distribute just to paying people. For example, if you have a class Moodle/Blackboard installation, you may not be allowed to put actual copies of the books up there. (I’m not sure of the details, but that would be worth investigating.) You can, however, put links to the books there. You can also host your own copies of the books if you want, as long as they’re available for free. That might mean that you put up copies of the books on your personal section of the school’s website, on your library’s website, or some similar resource.

    Kay Dutz: That is concerning indeed. I’ll contact you via email to get more information, but I’ll definitely want to correct the errors. Thank you for letting me know!

    Comment by Andy — 12/5/2014 @ 5:53 pm

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

1/16/2013

Flat World Knowledge

Filed under: Education, School — Andy @ 10:00 am

Over the past few years, a publishing company called Flat World Knowledge has been publishing a number of textbooks in several subject areas, from history to psychology to math. One of the features they have advertised is their “open” books, meaning in part that their books are available for free online to everyone. Until recently, this was nearly unheard of: students can now legally get their textbooks for free (while paying for extra features if they want them). While I had not heard about their books until recently (likely because they have few math books), this is definitely something I like, at least in the abstract.

Unfortunately, Flat World Knowledge has recently decided that the “open” model will not work for their publishing, because not enough people were buying their books. As much as I would like to argue that such a model should work, I’m sure they have more data than I do, and have undoubtedly done their analysis and decided that such a business model is unsustainable for them at this time. While I hope that they are able to offer their books in an open manner again in the future, they have at this point decided to restrict the way in which their books are available on their website, starting on January 1, 2013. (They have already started implementing this change, as well.)

The good news is that they previously published their books online under a Creative Commons license, a common license which allows redistribution (in particular, the attribution, share-alike, non-commercial license, version 3.0). This means that people have the right to continue to redistribute copies of the books, if they happen to have them.

I am still a bit disappointed: I would have liked Flat World Knowledge to succeed in their open publishing experiment. I would have liked more books to be available, and I would have liked even more companies to follow in their footsteps. Unfortunately, it appears as though that area may remain the realm of private or government financing for the moment.

I would like to remark for the inevitable debates to ensue in unseen boardrooms in the future that the Creative Commons license likely allowed Flat World Knowledge to have so many books. In nearly every foreword I had read, the authors extolled the open license of the book as a primary reason for publishing with FWK. Were it not for this license, it is entirely probable that FWK would not be in such a favorable position. The ability of others to share your books should be regarded as the feature so many authors see it as, rather than a liability.

14 Comments »

  1. Andy, I just found a chapter from my not-yet-published textbook on your website. One of the reasons I contacted Flat World Knowledge first when I was ready to look for a publisher was their policy of providing free online copies of the textbooks. I was very disappointed when I learned they would no longer provide them.

    I have one problem with the material you have online. The picture with the ThinkStock watermark does not belong to me and is not in the public domain. Would you please remove this picture before we both hear from ThinkStock’s attorneys? If the picture makes it into my textbook, it will be because FlatWorld has a contractual agreement with ThinkStock that gives them permission to use it. The rest of the images are mine or are in the public domain.

    Thank you.
    Carol A. Lowe, Ph.D.

    Comment by Dr. Carol A. Lowe — 1/27/2013 @ 1:31 pm

  2. Hi Andy! I just wanted to thank you for archiving the FWK texts. I’m an OER advocate and OER course developer for http://www.lumenlearning.com/ Do you mind if I link to your FWK archive? Much thanks for your good work!

    Comment by Ronda Neugebauer — 5/14/2013 @ 12:58 pm

  3. Is it O.K. to reuse specific figures from the textbooks? It would seem that having been released into the common domain, they couldn’t be reclaimed.

    Comment by Steve Edwards — 7/20/2013 @ 5:45 pm

  4. Thank you so much! This is very useful.

    Comment by Jenny Huy — 7/21/2013 @ 1:03 am

  5. Dr. Carol A. Lowe: Sorry for taking so long to get back to you on this. I believe I’ve finally found and replaced all of the ThinkStock watermarked photos in your Survey of British Literature book. In a few cases, the photo that was used was easily replaced with another public domain copy of the source image (such as the Armada Portrait of Queen Elizabeth I of England). In other cases, I had to replace the pictures with an image saying that the image was permanently unavailable. If you find other cases where there is a ThinkStock image, please let me know and I’ll be sure to get it replaced. Thank you for letting me know!

    Ronda Neugebauer: Thanks for the work that you do! You are of course always welcome to link to any part of the archive. (You can also download any of the books, copy it wholesale, or whatever you’d like within the bounds of the CC by-nc-sa license. In particular, though, if you do re-host them, please make sure that you aren’t violating the noncommercial aspect of the license.)

    Steve Edwards: As far as I know, it is fine to reuse any of the figures from the books. They were distributed according to the Creative Commons by-nc-sa 3.0 license, which means that you can copy any part of them as long as you credit the original author (if they allow you to do so), don’t use it for commercial purposes, and share your copy under the same terms. I don’t know what your purposes are, but I suspect that as long as you’re not selling the result or making money because of it, you would be fine to copy the figures. In addition, Flat World Knowledge has asked me to not label my redistribution of their books with their name, and I would expect that such a request would also apply to you: that means that you in fact should not credit Flat World Knowledge or the original authors. (While I find their request strange, it is what they have asked, so I’m passing that along.)

    In general, if you would like to reuse any part of these books, and it doesn’t appear as though someone has slipped up in clearing material (such as the ThinkStock watermarks Dr. Lowe pointed out), you are likely fine to use the material as long as you aren’t making money off of them, and your copies are also available under the same license. (Note that if you include the figures in a larger work, generally the larger work does not have to be available under the same license.) However, if you are concerned about the legality in your situation, you should talk to a lawyer. I’m not a lawyer, and my suggestions are based only on my best understanding, and are not legal advice.

    Jenny Huy: Thank you!

    Comment by Andy — 7/28/2013 @ 10:09 pm

  6. Thank you so much for maintaining these valuable open-ed resources!

    Comment by Jessica — 7/31/2013 @ 2:19 am

  7. Hi again, Andy-
    Faculty and students LOVE the 2012 Book Archive UI for the open texts. Are you interested in adding any new texts to your listing? I was sent a pdf of a January 2014 CC BY NC SA Financial Accounting text. Many thanks for your open contributions!
    Ronda

    Comment by Ronda Neugebauer — 1/22/2014 @ 2:02 pm

  8. Jessica: Thank you!

    Rhonda: Thanks for the kind words. I think I’m going to keep the archive you’re referring to as-is, just to keep that collection as one unit (particularly given its name), but I’d be happy to get a copy of the other text you’re referring to. I’ll have to see about setting up a generalized Creative Commons textbook reference page. I know that some places (such as Saylor and the OER Consortium) also have such lists, so they may be interested as well. I’ll contact you via email to inquire about getting a copy. Thanks for thinking of me!

    Comment by Andy — 1/22/2014 @ 4:51 pm

  9. I am writing to inform you that the use of our images in Principles of General Chemistry (v. 1.0) eISBN 978-1-4533-3122-4 and (v.1.0M ) eISBN 978-1-4533-5069-0 is unauthorized and is an infringement of our copyright.
    Fundamental Photographs is a studio and photo library specializing in science illustration. When CHEMISTRY: Principles, Patterns, and Applications 1e by AVERILL and Eldredge was published by Pearson Science, many of the copyrighted photographs were created and licensed to Pearson for use in that edition. When Pearson handed the files to the author, he was informed in writing that licensing for any future use was required for the images and illustrations in the text.
    In 2012 we discovered that our images were being provided unlawfully and without license in a digital and b/w version of the text by Flat World on Creative Commons. There is no legal Creative Commons license for this material. We reached an agreement with Pam Hersperger and she proceeded to remove the title from Creative Commons and license usage from us.
    I now find that this project with our photographs has been “rescued”, apparently without questioning why it had to be removed from Creative Commons. Your knowledge sharing goals are praiseworthy, but our photographs are created by hard-working individuals, oftentimes under demanding and volatile conditions, and they deserve to be properly remunerated. In posting our images without our permission, you invite third parties to continue to post this copyrighted material, without creator consent or reimbursement.
    We appreciate your removing these unlawfully posted images from your site and we look forward to your confirmation of this.
    Sincerely,
    Kip Peticolas, Partner
    Fundamental Photographs

    Comment by Kip Peticolas — 3/19/2014 @ 10:31 am

  10. Hi Kip,

    Thanks for contacting me. I am certainly aware of the difficulty involved in making high-quality photographs for educational use, and had presumed that, as required by the Creative Commons license, the publisher had either produced the images in their books or otherwise properly renumerated the photographers for their work. From what you say, that doesn’t appear to have always been the case.

    I’m truly sorry that you had this experience, and am committed to making it right. I’d be happy to work with you to remove the images which are not available under Creative Commons. I’ll contact you via the email address you provided to work out the details.

    Thanks again,
    Andy Schmitz

    Comment by Andy — 3/19/2014 @ 8:34 pm

  11. I’m not able to access the textbooks that were previously available. Can you help? Thank you!

    Comment by Sheri — 8/21/2014 @ 2:28 pm

  12. (For reference: the issue with Fundamental Photographs was resolved relatively easily. Their photographs are no longer in the books, because they have concerns about them being relicensed. Nonetheless, Kip was quite pleasant to work with, and I would not hesitate to recommend working with them if you do need chemistry photographs.)

    Sheri: I’ll contact you via email to verify that the issue has been resolved.

    Comment by Andy — 12/5/2014 @ 5:59 pm

  13. Andy,

    I understand that these books from flatworld were licensed under creative commons but that was done under a business model that also provided revenue to the authors of the books.

    New CEOs at flat world went with a different model in 20102 which involves fair-priced, but not free, textbooks.

    As an author of one of the books that you are hosting, I would ask that you do the right thing and remove the books from your site and ask your users to pay the fair price for the texts — this is about $25 per book.

    To keep the books posted is legally permissible but in my morally questionnable — it is unfair to the books’ authors.

    Thank you.

    Comment by Charles Stangor — 2/10/2015 @ 3:05 pm

  14. Charles Stangor: I’m sorry that the current situation isn’t to your liking. I have had a number of authors contact me and thank me for keeping the books freely available, including at least one above on this page.

    I’m not privy to the details of your agreements with Flat World Knowledge, but my guess was that authors received revenue when other forms of their books (or additional resources) were purchased, not when people simply viewed the free versions online. In that case, the copies I’ve hosted online are essentially the same as what had been available before.

    Previously, my copies of the books had been hosted with full links to the original book’s website, and I had encouraged readers to support the authors and publisher by purchasing the book or additional resources. Unfortunately, the publisher contacted me and asked me to remove all attribution, so those links are no longer available. (I would be quite happy to restore these links if I received such a request from the publisher, of course.)

    I can certainly appreciate the fact that the books are available for fair prices, and that this is a significant improvement over the status quo. I have found that a significant portion of the visitors to my copies of books are from other countries (frequently, India) where even arranging to pay the publisher at all might prove quite difficult. (The difficulty is pure speculation, as I’m not in that situation, but it seems plausible.)

    I understand your position and concern over potential lost revenue. On balance, I still think it is better to keep the content available to more people, and I’m sorry that we disagree.

    Thank you for commenting,
    Andy Schmitz

    Comment by Andy — 2/15/2015 @ 1:24 pm

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

12/25/2012

Merry Christmas, Again

Filed under: General — Andy @ 12:00 am

Another year has passed, and with just one [other] published post here. I’m slowly starting to have more discretionary time, and to get some projects closer to completion. Barring any major surprises, I should have at least one or two things to release next year. (The problem, of course, with long-running projects is that there are few of them, and if I should decide for whatever reason to not release them, they tend to disappear entirely.)

In the arbitrary milestones department, GraphSketch passed 1.5 million graphs delivered a while back, and I should really pay attention for the 2 million milestone that’ll be coming up sooner or later. I’ve also obtained a full-time job, which surprisingly seems to take less time than school did at times. It seems to be pretty nice, although, as with many things, I haven’t yet taken the time to step back and thoroughly examine how it’s going. There will probably be more comments on the job in the future, but for the moment, allow me to disclaim everything posted here and say that unless an exception is clearly noted in a post, the posts on this blog represent only my own views (if that), and have never been, and will never be intended to represent the opinions of my employer, school, friends, colleagues, or anyone else.

So, this makes for a particularly lackluster sometimes-annual update, but it’s about everything I have to say. Have a Merry Christmas if that’s your thing, and have a great new year when that rolls around. With any luck, I’ll have something new to post in January.

Until then,
Andy Schmitz

No Comments »

No comments yet.

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

9/1/2012

Getting Complicated, Embedded YouTube Videos

Filed under: Hacks, School, Technology — Andy @ 3:29 pm

So, earlier today, Karl Fisch asked for a copy of an Olympics video from NBC’s website to use in his school. A number of the “standard” ways of getting the video didn’t seem to work, so I figured I’d pitch in. By the way, the first thing to try (if whatever addons you might have to download YouTube videos don’t work – although I don’t use any myself, I’m told they didn’t work in this case) is to click the YouTube button.

The YouTube button will usually open the video up into YouTube’s website, where most addons for downloading a video will work better, and there are plenty of other websites happy to help you.

In this case, clicking the button only got to the NBC Olympics home on YouTube. A quick search of the channel showed that the video I was looking for wasn’t on the NBC Olympics channel, and I’d have to find another way to download it.

(more…)

2 Comments »

  1. Hi,
    I am trying to built a website where user can download Youtube videos. I have really worked very hard to learn how to code in PHP and get videos from Youtube.

    I am getting all the information about the video with this function:
    parse_str(file_get_contents(‘https://www.youtube.com/get_video_info?video_id=’.$id),$info);

    But the problem is when I refresh the page sometimes I get the “quality” and “type” parameter’s values and sometime not.

    What it going on please help me.

    Comment by Kelly — 6/8/2015 @ 2:02 am

  2. Kelly: Unfortunately, I can’t really help there. You might take a look at the code from youtube-dl, which is generally considered to be one of the better tools for downloading YouTube videos. The video I’m discussing didn’t work with youtube-dl back when I wrote the post, but it probably does now: the youtube-dl developers definitely keep up with those changes much better than I can.

    Comment by Andy — 10/26/2015 @ 5:04 pm

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

4/4/2011

Portrait: Linux System Management Tool

Filed under: General — Andy @ 3:13 pm

So, over the past year or so, I’ve been working on a project I’m calling Portrait. It’s [going to be] a piece of open-source software system administrators can use to manage Linux computers. Unfortunately, there’s still a fair amount of work to be done on Portrait, because my free time is pretty sparse. I’d also like to launch a hosted version of Portrait, which people can use if they don’t want to deal with setting up and maintaining a copy of the software. To help get both of those things moving, I’ve set up a Kickstarter project, titled simply Portrait: Linux System Management Tool to raise funds to let me work on Portrait over the summer and set up a hosted version that people can use.

Most of my readers won’t personally have any use for a Linux system management tool, but if you happen to know anyone you think might find it useful, I’ll ask you to please pass the Kickstarter project along to them. If you’re interested yourself, please head on over to the Kickstarter project page, where I’ve put much more information on exactly what Portrait will do.

Andy Schmitz

5 Comments »

  1. Hi Andy,

    I found a link to your kickstarter project from a post you made on the ubuntu forum.

    I’m also looking at possibly building a solution to this land scape issue for patching our server park.

    On the kick starter site you posted an update stating that you had received an internship and so would not be working on Portrait any further this summer.

    As you intended to release the project as opensource under the GPL, why do you not post up the code which you have already written for review? I would most certinaly be interested in it.

    Kind Regards,
    Jamie.

    Comment by Jamie Mack — 7/29/2011 @ 5:08 am

  2. Hi Jamie,

    That’s a good suggestion. Unfortunately, my code at the moment is written in a number of different languages, and not particularly useful to most people in the current form. I do have a goal of cleaning it up and getting things to a usable point for more people, but I’d prefer to avoid releasing something that’s mostly a confusing set of different languages, hard-coded paths, and so forth. On the other hand, if you’d like to email me (andy dot schmitz at gmail dot com), I can certainly try to explain what I’ve found and been able to work with in terms of communication with Landscape.

    As it is, Portrait is something that I’d like to at least slowly advance, so it’s something I expect to be working with in my spare time at school. It’s possible that a proper initial release will in fact be available within the next few months.

    Thanks,
    Andy Schmitz

    Comment by Andy — 7/30/2011 @ 4:21 pm

  3. Hi,

    I noticed that last year May you cancelled this project, is it still on the table?

    Thanks;

    Andrew

    Comment by Andrew — 11/18/2012 @ 8:39 pm

  4. Is there anything still going on with Portrait? Would be really cool if you’d be able to get this ready for use 🙂

    /Paul

    Comment by Paul — 2/5/2013 @ 2:21 pm

  5. Andrew and Paul: Unfortunately, I’ve been busy enough with my other projects to not really have time to get Portrait to a better state. I’m currently mostly stuck on finding a good backend for storing historical data. RRDtool seemed to be the popular option for a while, and Graphite seems to be becoming more popular, but it’s unfortunately pretty painful to set up. If anybody has ideas about other potential backends, they’d undoubtedly be useful.

    Comment by Andy — 3/3/2013 @ 8:03 pm

RSS feed for comments on this post.

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Next Page »
My Stuff
Blog Stuff
Categories
Archives