1. New Smogon apparel available for purchase!
  2. Welcome to Smogon Forums! Please take a minute to read the rules.
  3. Smogon's Official Tiering Policy Framework
  4. Click here to ensure that you never miss a new SmogonU video upload!
Smogon Apparel Global Rules Smogon's Official Tiering Policy Framework New Video Uploads

Old album scraping scripts

Discussion in 'Technical Projects' started by chaos, Sep 28, 2013.

  1. chaos

    chaos
    is a member of the Site Staffis a Battle Server Administratoris a Programmeris a Smogon IRC SOPis a Contributor to Smogonis an Administratoris a Tournament Director Alumnusis a Researcher Alumnus
    Owner

    Joined:
    Dec 18, 2004
    Messages:
    10,572
    These are for sandshrewz and Hugendugen, who are working on a new album.

    I have no idea who wrote these or how they work.

    Attached Files:

    chaos, Sep 28, 2013
    #1
    Kingler12345 likes this.
  2. sandshrewz

    sandshrewz POTATO
    is a Site Staff Alumnusis an Artist Alumnusis a Forum Moderator Alumnusis a Contributor Alumnusis a Smogon Media Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    Oct 18, 2010
    Messages:
    2,743
    chaos / Hugendugen I tried making it using JavaScript / jQuery. Turns out that getting the images spams with a lot of GET request idk why :( even though I'm using regex to get the image src attribute now .-. stuff not included: ignoring images in quote tags because people can quote from other people and mess it up.

    Currently it loads 10 pages worth of images and everytime you click the button it loads 10 more. Not sure what can be done about the image stuff without flooding with GET requests sooo :( anyway it's in http://www.smogon.com/album/albumtest might lag your internet a bit. (It brought me to 1.0Mbps when I used it fsr ._. maybe too spammy) !_! <_< >_> ^_^

    Edit: did nexus edit those smileys in...
    Last edited: Sep 28, 2013
    sandshrewz, Sep 28, 2013
    #2
  3. Toast++

    Toast++ CaTOASTrophe
    is a member of the Site Staffis a Battle Server Administratoris a Programmeris a Super Moderatoris a Researcher Alumnusis a Smogon Media Contributor Alumnus
    Super Moderator

    Joined:
    Mar 9, 2009
    Messages:
    2,275
    I brought the massive amount of GETs to sandz' attention (~200-500 per load) and I'd be happy to look at it to try to filter that stuff out.

    When the new forums were first put up, I could access a json feed for just about anything. Now I get a security error. This would, of course, help a ton with this project and at least a few others. Is it possible to bring that back?



    Edit: This was resolved. We're down to like ~30. If there were a services, this could be improved.
    Last edited: Oct 5, 2013
    Toast++, Sep 28, 2013
    #3
  4. Shiv

    Shiv mostly harmless
    is a Site Staff Alumnusis a Smogon IRC AOp Alumnusis a Programmer Alumnusis a Forum Moderator Alumnusis a Battle Server Moderator Alumnusis a Past WCoP Champion

    Joined:
    Apr 7, 2005
    Messages:
    5,876
    I think Brain wrote them and I added stuff in if I remember right fyi.

    Weren't the best written stuff tbh, but it did the job (atleast back then).
    Shiv, Oct 15, 2013
    #4

Users Viewing Thread (Users: 0, Guests: 0)