1. Welcome to Smogon Forums! Please take a minute to read the rules.
  3. Try out our new beta Pokedex.
  4. New to the forums? Check out our Mentorship Program!
    Our mentors will answer your questions and help you become a part of the community!
Rules Badge FAQ New Dex Mentorship

Old album scraping scripts

Discussion in 'Technical Projects' started by chaos, Sep 28, 2013.

  1. chaos

    chaos
    is a member of the Site Staffis a Battle Server Administratoris a Programmeris a Smogon IRC SOPis a Contributor to Smogonis an Administratoris a Tournament Director Alumnusis a Researcher Alumnus
    Owner

    Joined:
    Dec 18, 2004
    Messages:
    9,488
    These are for sandshrewz and Hugendugen, who are working on a new album.

    I have no idea who wrote these or how they work.

    Attached Files:

    chaos, Sep 28, 2013
    #1
    Kingler12345 likes this.
  2. sandshrewz

    sandshrewz 缘
    is a Site Staff Alumnusis an Artist Alumnusis a Forum Moderator Alumnusis a Smogon Media Contributor Alumnusis a Contributor Alumnusis a Battle Server Moderator Alumnus

    Joined:
    Oct 18, 2010
    Messages:
    2,428
    chaos / Hugendugen I tried making it using JavaScript / jQuery. Turns out that getting the images spams with a lot of GET request idk why :( even though I'm using regex to get the image src attribute now .-. stuff not included: ignoring images in quote tags because people can quote from other people and mess it up.

    Currently it loads 10 pages worth of images and everytime you click the button it loads 10 more. Not sure what can be done about the image stuff without flooding with GET requests sooo :( anyway it's in http://www.smogon.com/album/albumtest might lag your internet a bit. (It brought me to 1.0Mbps when I used it fsr ._. maybe too spammy) !_! <_< >_> ^_^

    Edit: did nexus edit those smileys in...
    Last edited: Sep 28, 2013
    sandshrewz, Sep 28, 2013
    #2
  3. Toast++

    Toast++ Butter Included.
    is a member of the Site Staffis a Programmeris a Super Moderatoris a Smogon Media Contributoris a Researcher Alumnus
    Programmer

    Joined:
    Mar 9, 2009
    Messages:
    1,603
    I brought the massive amount of GETs to sandz' attention (~200-500 per load) and I'd be happy to look at it to try to filter that stuff out.

    When the new forums were first put up, I could access a json feed for just about anything. Now I get a security error. This would, of course, help a ton with this project and at least a few others. Is it possible to bring that back?



    Edit: This was resolved. We're down to like ~30. If there were a services, this could be improved.
    Last edited: Oct 5, 2013
    Toast++, Sep 28, 2013
    #3
  4. Shiv

    Shiv mostly harmless
    is a Site Staff Alumnusis a Smogon IRC AOp Alumnusis a Forum Moderator Alumnusis a Battle Server Moderator Alumnusis a Past WCoP Winner

    Joined:
    Apr 7, 2005
    Messages:
    5,870
    I think Brain wrote them and I added stuff in if I remember right fyi.

    Weren't the best written stuff tbh, but it did the job (atleast back then).
    Shiv, Oct 15, 2013
    #4

Users Viewing Thread (Users: 0, Guests: 0)