Databases

(SilverEdge/55442) BWS> That's what I was leaning towards. However, I'm trying to ...

SilverEdge@rss.iscabbs.com — Mon, 20 Feb 2012 09:37:00 -0600

BWS> That's what I was leaning towards.  However, I'm trying to make this as
simple as possible for my users.  They aren't too tech savy.  As it is, it
might become a moot point as my supervisor asked about an interface for these
files, so that they wouldn't have to worry about putting Excel into window mode
to view both of them - again, to make it easier on the non-tech-savy users.

Thank you anyway for the workaround though.  :)

(Boogie With Stu/55441) SilverEdge: our way around that Excel-leading quote business wa...

Boogie With Stu@rss.iscabbs.com — Sun, 19 Feb 2012 03:11:00 -0600

SilverEdge:  our way around that Excel-leading quote business was to always
export the fields with a leading-quote character so stoopid Excel wouldn't
strip our leading zero characters in text fields, and then make the end user
do a find-and-replace (ctrl-H on Windows) in Excel, finding any single-quote
character and replacing it with....a single-quote character.  Like magic,
they all disappear (after Excel iterates over every cell in the worksheet),
and the leading zeroes are intact.

(Kraven/55439) Try this, '="'+VALUE_WITH_LEADING_ZEROS_HERE+'"'

Kraven@rss.iscabbs.com — Tue, 14 Feb 2012 12:26:00 -0600

Try this, '="'+VALUE_WITH_LEADING_ZEROS_HERE+'"'
I was just having that same problem and now its gone, though i guess it may
cause problems with formulas

(SilverEdge/55437) Doing a query that will export into a CSV to be viewed in Excel....

SilverEdge@rss.iscabbs.com — Tue, 14 Feb 2012 11:05:00 -0600

Doing a query that will export into a CSV to be viewed in Excel.  This will run
automatically and the user will open it as needed.  Unfortunately, a couple of
the fields are IDs, and sometimes have a 0 in front of the number.  Excel by
default makes them go away, but we need to see them.  I modified the SQL query
to be '''' + A1.PI_ID, so that the result is something like '012345.  My hope
was that Excel would treat it as text.  Well, it does.  However, it still
displays the '.  Odds are it is just displaying the ' and the ' makes it no
longer a number.  What can I do to stop this?

(BDavis/55436) Actually that is not Oracle-specific, and I think is part of som...

BDavis@rss.iscabbs.com — Mon, 06 Feb 2012 09:30:00 -0600

Actually that is not Oracle-specific, and I think is part of some version of
ANSI SQL. So it should work for most databases, and yes it is useful. You can
extend it, e.g. "order by 3,1,2,4". Also useful for calculated columns that
even if they are named can often not use the name/alias.

(LongTimeLurker/55435) Fav 'lil Oracle SQL shortcut:

LongTimeLurker@rss.iscabbs.com — Mon, 06 Feb 2012 08:45:00 -0600


Fav 'lil Oracle SQL shortcut:

If you are selecting from a table and you want to sort the output by
'super_huge_long_column_name' and that is the 4th column in the output then you
can go:  Select * from MyTable order by 4;

schweet :-)

(BDavis/55434) lol - I know what you mean. I think many of us just pick up a to...

BDavis@rss.iscabbs.com — Mon, 06 Feb 2012 05:51:00 -0600

lol - I know what you mean. I think many of us just pick up a tool and run with
it without really learning all the features. I just learned some similar
shortcuts with Oracle's PL/SQL, though by watching someone not a class.

(Stilgar/55433) So I'm in SQL Server training this week. I hate it when I learn...

Stilgar@rss.iscabbs.com — Mon, 06 Feb 2012 05:45:00 -0600

So I'm in SQL Server training this week.  I hate it when I learn about dumb
shit that I should have known all along.  I just learned that in SSMS, you can
drag db, table, column, etc names into the query window rather than typing all
that shit.

(Danix/55432) BDavis> Let's just say that a full filesystem made me turn into ...

Danix@rss.iscabbs.com — Mon, 30 Jan 2012 07:09:00 -0600

BDavis> Let's just say that a full filesystem made me turn into an SQL Server
expert overnight. It also led to the people responsible for switching the
recovery model to "Simple" on that particular production database getting axed.

(BDavis/55431) Yes, you can recover with transaction logs. But if you've never ...

BDavis@rss.iscabbs.com — Mon, 30 Jan 2012 07:02:00 -0600

Yes, you can recover with transaction logs. But if you've never done that it
can require a lot of explanation, so I was trying to keep it simpler. :-P
And yes it would assume you were using a full recovery model. It is also
often possible to recover without an intact log file. As with many computer
things, it just depends how important it is and how much expertise you have.

(Danix/55430) BDavis> Theoretically, the transaction logs should be able to fi...

Danix@rss.iscabbs.com — Mon, 30 Jan 2012 05:34:00 -0600

BDavis> Theoretically, the transaction logs should be able to fix the DB if it
was shut down in an "unclean" state, shouldn't it? Unless they used the
"Simple" recovery model, which would mean they're screwed :(

(BDavis/55429) The only problem with reattaching the database files would be th...

BDavis@rss.iscabbs.com — Sat, 28 Jan 2012 20:41:00 -0600

The only problem with reattaching the database files would be that it probably
did not get a clean shutdown. So if restoring from backup is acceptable that is
likely a safer choice. You may be fine with the reattach, but I would check the
database and server logs thoroughly afterwards, run integrity checks, etc. to
be sure you don't end up with something out of whack showing up later.

(Gooseman/55428) If you have a backup of the DB, you should be able to restore it...

Gooseman@rss.iscabbs.com — Sat, 28 Jan 2012 17:18:00 -0600

If you have a backup of the DB, you should be able to restore it once SQL is
reinstalled.  I've moved DBs that way before.  I hope, though, that it's a real
DB backup, and not just a backup of the DB files - that doesn't always work as
well.

(Keogk/55427) I have a database riunning on a MS sql server 2008.

Keogk@rss.iscabbs.com — Sat, 28 Jan 2012 16:53:00 -0600

I have a database riunning on a MS sql server 2008.
The main HD failed and the raid also failed so the OS was lost.
The drive with the database is fine and I have a backup of the database as well
Can I just reinstall the OS on new drives. Give the server the same name and IP
 reinstall sql and attach the database again. Or will I need to do more ?

(Smoke Eater/55426) no, it cannot. not enough data fields. Tried that first, and A...

Smoke Eater@rss.iscabbs.com — Tue, 17 Jan 2012 16:19:00 -0600

no, it cannot.  not enough data fields.  Tried that first, and Access calls
them crosstabs.  I found something that works, but thanks.

(BDavis/55425) Smoke Eater> I think what you want can also be done with pivot q...

BDavis@rss.iscabbs.com — Tue, 17 Jan 2012 16:15:00 -0600

Smoke Eater> I think what you want can also be done with pivot queries, if your
database supports it. Just a thought.

(Smoke Eater/55424) http://allenbrowne.com/func-concat.html

Smoke Eater@rss.iscabbs.com — Tue, 17 Jan 2012 10:47:00 -0600

http://allenbrowne.com/func-concat.html

Please disregard my previous post.  Not two minutes after I posted it, though
I'd been searching for a while, I found the above link, that led me to a VBA
function called ConcatRelated(), and that solves my problem!

(Smoke Eater/55423) I'm having a serious brain cramp here....I know I used to know...

Smoke Eater@rss.iscabbs.com — Tue, 17 Jan 2012 10:19:00 -0600

I'm having a serious brain cramp here....I *know* I used to know how to do
this.

I have a table with two fields: CaseNumber and TextData (not the real fields,
but okay).

Can anyone come up with a query that will give me all of

CaseNumber TextData TextData TextData?

There should be as many TextData fields in there as there are for that record,
and the rest can be blank.


For example, if I have this:

CaseNumber     TextData
1              ABC
1              DEF
1              GHI

I want this:

CaseNumber     TextData   TextDAta   TextData
1              ABC        DEF        GHI

(Raubvogel/55422) I want to have an user in mysql that can only run show slave sta...

Raubvogel@rss.iscabbs.com — Tue, 17 Jan 2012 08:34:00 -0600

        I want to have an user in mysql that can only run show slave status.
I take that a "grant show slave status" would work, but on which database?

(Boogie With Stu/55421) Oracle's MERGE INTO syntax is quite handy and flexible for doing...

Boogie With Stu@rss.iscabbs.com — Sun, 15 Jan 2012 19:06:00 -0600

Oracle's MERGE INTO syntax is quite handy and flexible for doing that UPSERT
stuff in just one statement.

(JL/55420) To be database-neutral, you do an update, and if there were <1 r...

JL@rss.iscabbs.com — Sun, 15 Jan 2012 16:40:00 -0600

To be database-neutral, you do an update, and if there were <1 rows updated,
you do an insert.

Some databases support things like UPSERT which does either update or insert.

(Kena/55419) What's the "right" way to, given a value, do either an insert or...

Kena@rss.iscabbs.com — Sun, 15 Jan 2012 12:17:00 -0600

What's the "right" way to, given a value, do either an insert or an update, as
appropriate (based on whether or not the corresponding record already exists)?

(Smoke Eater/55418) Is there anyone online now who is an Oracle guru? If so, please...

Smoke Eater@rss.iscabbs.com — Wed, 11 Jan 2012 08:07:00 -0600

Is there anyone online now who is an Oracle guru?  If so, please X me.

(John Public/55417) InnoDB, usually, but other engines as well.

John Public@rss.iscabbs.com — Mon, 09 Jan 2012 15:59:00 -0600

InnoDB, usually, but other engines as well.

(River Rat/55416) JP, what engine are you running? InnoDB or the other one?

River Rat@rss.iscabbs.com — Mon, 09 Jan 2012 13:19:00 -0600

JP, what engine are you running?  InnoDB or the other one?

(John Public/55415) My preference is to push backups to a drop location, then allow ...

John Public@rss.iscabbs.com — Mon, 09 Jan 2012 11:21:00 -0600

My preference is to push backups to a drop location, then allow the backup
system to pull from the drop. It separates the creation of the backup from the
archiving of the backups themselves. And it avoids exposing ssh/rsync access
deeper into the network.

Would this work for your case?

    1. cronjob on DB server pushes to the drop location:

        mysqldump -h localhost | gzip -1 > tempfile
        mv tempfile /drop/`date +%F`.sql.gz


    2. cronjob on backup server pulls from the drop location:

        rsync -a --ignore-existing backup@dbserver:/drop/  /backups/dbserver/

The --ignore-existing flag is there so a compromised dbserver can't overwrite
prior backups. The `mv` is there so you don't introduce a race condition
between the push and pull.

(Raubvogel/55414) I have a mysql server behind a webserver on the external

Raubvogel@rss.iscabbs.com — Mon, 09 Jan 2012 10:41:00 -0600

        I have a mysql server behind a webserver on the external
network. In our internal network I have the backup server. Which would
be the smartest/most secure way to backup the server:

 1) Have db server connect to a port in the firewal, which is forwarded
to the backup server, and then rsync/whatever the db dump there.

 2) Have backup server connect to a port in webserver which is then
forwarded to db server. Now it can either then do a dump or grab a dump
already waiting for it.

One dump a day if it matters.

(John Public/55413) Or `show create view NAME`. Either way, you get a shitty output....

John Public@rss.iscabbs.com — Wed, 28 Dec 2011 06:06:00 -0600

Or `show create view NAME`. Either way, you get a shitty output. :)

(Kena/55412) Ah. SELECT view_definition from information_schema.views where ...

Kena@rss.iscabbs.com — Wed, 28 Dec 2011 05:43:00 -0600

Ah.  SELECT view_definition from information_schema.views where table_name=foo;

(Kena/55411) I'm stuck. How do I see how a view is constructed? (MySQL) "D...

Kena@rss.iscabbs.com — Wed, 28 Dec 2011 05:31:00 -0600

I'm stuck.  How do I see how a view is constructed?  (MySQL)  "DESC <viewname>"
just shows me the table properties, not how it was derived.

(BDavis/55410) As I said, using the system catalogs it is certainly possible, I...

BDavis@rss.iscabbs.com — Tue, 27 Dec 2011 10:35:00 -0600

As I said, using the system catalogs it is certainly possible, I just do not
know of something you can download where someone already did the grunt work.

(The After Party/55409) BDavis

The After Party@rss.iscabbs.com — Tue, 27 Dec 2011 06:03:00 -0600

BDavis

  There has to be a way .. you can search all of the tables for column names,
  so it wouldn't seem to far off to then use those column names
  to query ..

  I found the other table i was looking for so i'm not going to mess
  around with a long running query who's synta i don't know.

(BDavis/55408) If you are desparate enough you might enable Oracle Text, though...

BDavis@rss.iscabbs.com — Tue, 27 Dec 2011 05:40:00 -0600

If you are desparate enough you might enable Oracle Text, though that seems
like overkill. I don't know of anything that walks the system catalogs to do a
search, but it could be written.

As far as database models, as far as I know Oracle does not offer anything for
free or built-in to the DBA tools. There are lots of 3rd party tools to reverse
engineer a database. Many rely on referential integrity being defined in the
database, which a shockingly large number of commercial apps do not define. The
ones I am familiar with are not exactly cheap, though Google reveals a number
of free ones that might be enough to get you through this. Just do a search for
"database design reverse engineer free".
This one looked interesting if basic.  http://www.dbwrench.com/index.shtml

(The After Party/55407) Oracle 11g, record search

The After Party@rss.iscabbs.com — Tue, 27 Dec 2011 04:10:00 -0600

Oracle 11g, record search

  Is there SQL (or stored proc) that can search all tables for any rows
  that contain a specific string? Yeah, I know that coudl take awhile.
  Its a non-production environment. Somehow, there must be an entry in
  the DB pointed to some old location, and I'd like to update it to
  to point to the new location. The ecomerce server (ATG) is a freaking
  nightmare of tables (well, at least for this non-dba, me). I'm sure
  a table relationship diagram would help (don't know where it would be)
  as I found one table with the old values and corrected it, but that didn't
  sovle the problem. arg.

(Banshee/55406) DOH! My bad too. There were actually two errors I should have ...

Banshee@rss.iscabbs.com — Thu, 22 Dec 2011 11:30:00 -0600


DOH!  My bad too.  There were actually two errors I should have seen for
myself.  The one being reported was the lack of a comma before the SUM.  And
then as already spotted, the trailing parenthesis moved to right after the END.

Works great!  Thanks.

(Smoke Eater/55405) smacks forehead Of course!!!! Sorry Banshee, I should have c...

Smoke Eater@rss.iscabbs.com — Thu, 22 Dec 2011 09:21:00 -0600

*smacks forehead*  Of course!!!!  Sorry Banshee, I should have caught that one.

(BDavis/55404) Yes that is 2005, http://www.sqlteam.com/article/sql-server-vers...

BDavis@rss.iscabbs.com — Thu, 22 Dec 2011 09:03:00 -0600

Yes that is 2005, http://www.sqlteam.com/article/sql-server-versions.
I think you have a misplaced closing parenthesis. Put it after the END instead
of after MNCount. (I did not test this.)

(Smoke Eater/55403) Possibly. I'll have to check on it....

Smoke Eater@rss.iscabbs.com — Thu, 22 Dec 2011 08:42:00 -0600

Possibly.  I'll have to check on it....

(Banshee/55402) Smoke Eater> I'm actually getting a "Incorrect syntax near keyw...

Banshee@rss.iscabbs.com — Thu, 22 Dec 2011 08:03:00 -0600


Smoke Eater>  I'm actually getting a "Incorrect syntax near keyword 'CASE'
error:

SELECT YOFFIC
   SUM( CASE YSUBCL
           WHEN 'M'
              THEN 1
           WHEN 'N'
              THEN 1
           ELSE 0
           END AS MNCount)
FROM Table.dbo.GNACMFF1
GROUP BY YOFFIC

This is SQL Server 9.0.4.229 -- which I think is 2005, no?

(Smoke Eater/55401) SFTT:

Smoke Eater@rss.iscabbs.com — Thu, 22 Dec 2011 07:59:00 -0600

SFTT:

So my query would have had teh following results

Category   AttributeCount
--------   --------------
bar              4
baz              0
foo              3

(Smoke Eater/55400) Now, my solution gets all the categories rather than products......

Smoke Eater@rss.iscabbs.com — Thu, 22 Dec 2011 07:58:00 -0600

Now, my solution gets all the categories rather than products...because that's
what your examples said.

Also, it assumes that everything is in the one table.

(Smoke Eater/55399) SELECT TableA.Category,

Smoke Eater@rss.iscabbs.com — Thu, 22 Dec 2011 07:55:00 -0600

SELECT TableA.Category,
   SUM( CASE Attribute
           WHEN 'A'
              THEN 1
           WHEN 'B'
              THEN 1
           ELSE 0
           END AS AttributeCount)
FROM TableA
GROUP BY Category

(JL/55398) How does the 'product' column come into play if you are only sum...

JL@rss.iscabbs.com — Thu, 22 Dec 2011 07:54:00 -0600

How does the 'product' column come into play if you are only summing up
attribute existences?

If you have multiple A attributes for the same product, should that only count
as 1 towards the total?

(Banshee/55397) MSSQL (2005, I think). I'm struggling with a DB query that can ...

Banshee@rss.iscabbs.com — Thu, 22 Dec 2011 05:23:00 -0600


MSSQL (2005, I think).  I'm struggling with a DB query that can return "zero
values" within a count/group by.  Here's what I've got;


    TABLE A
    category    product    attribute
    --------------------------------------------------
    foo         ABC        A
    foo         DEF        Z
    foo         GHI        A
    foo         JKL        B
    bar         MNO        A
    bar         PQR        B
    bar         STU        B
    bar         VWX        B
    baz         YZ1        C
    baz         234        C


Now let's say I want to get a count of all the products that have the
attributes
A or B in each category, including a tally of those that don't.  So the results
look like:

    category    count
    -------------------------
    foo         3
    bar         4
    baz         0


And conversely, a count of all products in each category that do NOT have an
attribute of A or B;

    category    count
    -------------------------
    foo         1
    bar         0
    baz         2


What am I looking for here?  I was thinking this is a left outer join, but I'm
getting tripped out when combining that with a count and group by clause.

(Yorick/55396) In Oracle, you may want one of two different things.

Yorick@rss.iscabbs.com — Tue, 20 Dec 2011 14:26:00 -0600

In Oracle, you may want one of two different things.
Flashback Data Archive lets you store a history table that lets you
recreate point-in-time results from a particular table.
http://uhesse.wordpress.com/2009/10/23/total-recall-brief-introduction-into-fla
shback-data-archive/

Auditing allows for records of various operations to be saved to an audit
trail.  There's no particular features for "time travel", but it logs
code and user changes as well.

I think you want the flashback data archive.  I tried to use it in 9i
and it was very clunky, but I suspect it is much better.

(JL/55395) After some reading and discussion, I'm going to recommend that m...

JL@rss.iscabbs.com — Tue, 20 Dec 2011 11:09:00 -0600


After some reading and discussion, I'm going to recommend that my client move
from Sybase ASE 15 to Oracle, for two reasons:

1. Materialized Views. Everyone has them, but not Sybase! This will allow me
   to get automatic aggregate data and increase performance.

2. A colleague said Oracle supports automatic "audit trail" functionalty to
   track changed values. We currently do this in the application code, along
   with triggers in the DB that populat an audit trail table.
   I'm just learning about this, but has anyone used this in Oracle?

(BDavis/55394) I have been working with databases long enough to have developed...

BDavis@rss.iscabbs.com — Sun, 18 Dec 2011 19:42:00 -0600

I have been working with databases long enough to have developed my own mental
library of design patterns based on experience. But the question was
interesting enough so I googled a bit and found this. Looks interesting.
  http://databaseanswers.org/data_models/index.htm

(JL/55393) So I'm now further exploring some of the database design questio...

JL@rss.iscabbs.com — Sun, 18 Dec 2011 18:59:00 -0600

So I'm now further exploring some of the database design questions I asked
earlier, and I'm getting  bit overwhelmed. I'm nowhere near a DB guru.

Where would I go to get "Design Patterns" for databases, for lack of a better
word?

I really would love to see real-world cases where people say "We had problem X
and Y, we needed Z, so we implemented solution ABC".

I'm not sure I'm qualified to be giving specific answers about how to best
implement a database/warehouse design, but it's on me, so I'm doing the best I
can ;)

(BDavis/55392) Database modeling usually refers to a using a tool to visually l...

BDavis@rss.iscabbs.com — Sat, 17 Dec 2011 10:14:00 -0600

Database modeling usually refers to a using a tool to visually lay out the
tables, columns, keys, indexes, relationships, etc. The tool then generates the
SQL for the target database(s) and/or creates it without you needing to do any
writing of DDL scripts. I would venture to say nearly everyone uses a modeling
tool of some kind if their job involves designing databases, and some have
embedded basic tools that let you do this. Capability beyond basic database
design vary a lot, and pricing is usually commensurate.

(River Rat/55391) Can someone elxplain what database modeling is?

River Rat@rss.iscabbs.com — Sat, 17 Dec 2011 03:44:00 -0600

Can someone elxplain what database modeling is?
I see it mentioned in mysql-workbench.
http://mysql.com/products/workbench/
But don't understand what it means.

(Raubvogel/55390) Replicate mysql question: how can I monitor the replication to

Raubvogel@rss.iscabbs.com — Sat, 10 Dec 2011 10:47:00 -0600

        Replicate mysql question: how can I monitor the replication to
make sure master and slave are in sync? And get an email/message/bird
with a candle when they are not?

(Boogie With Stu/55389) Later versions of Oracle suppport FULL OUTER JOIN syntax for tha...

Boogie With Stu@rss.iscabbs.com — Fri, 02 Dec 2011 22:53:00 -0600

Later versions of Oracle suppport FULL OUTER JOIN syntax for that purpose...I
think starting with 10g.

(John Public/55388) Which RDBMS? In MySQL, you might do,

John Public@rss.iscabbs.com — Thu, 01 Dec 2011 16:34:00 -0600

Which RDBMS? In MySQL, you might do,

    SELECT t1.account_id FROM t1 LEFT JOIN t2 USING(account_id)
        WHERE t2.account_id IS NULL
    UNION ALL
    SELECT t1.account_id FROM t1 RIGHT JOIN t2 USING(account_id)
        WHERE t1.account_id IS NULL

(Or use two left joins, and swap t1 and t2... same thing as far as MySQL
optimizations.)

(Smoke Eater/55387) But that only finds account IDs in table 1 tha aren't in table 2...

Smoke Eater@rss.iscabbs.com — Thu, 01 Dec 2011 16:52:00 -0600

But that only finds account IDs in table 1 tha aren't in table 2.  Do you also
want the reverse?  If so, you'll need the flip side of JL's query...and do a
UNION between those two queries....

(JL/55386) SELECT t1.account_id from table1

JL@rss.iscabbs.com — Thu, 01 Dec 2011 12:10:00 -0600


SELECT t1.account_id from table1
        LEFT OUTER JOIN table2 on table1.account_id=table2.account_id
        WHERE table2.account_id IS NULL

right?
But the subselect seems better to me.

(Egregious/55385) I want to find the account_id's that are different between two t...

Egregious@rss.iscabbs.com — Thu, 01 Dec 2011 11:49:00 -0600


I want to find the account_id's that are different between two tables.

I do it like this:

  SELECT account_id FROM table1 WHERE account_id NOT IN
    (SELECT account_id FROM table2);

And the reverse.

Is there a way to do that with JOINs that avoids a subselect?

(Cyanide/55384) JL> can you chop the data into "blocks" like 10k records, do the...

Cyanide@rss.iscabbs.com — Thu, 01 Dec 2011 07:36:00 -0600

JL> can you chop the data into "blocks" like 10k records, do the calculations
on that, and then put a trigger to do a partial aggregate update if a record
in that block changes. (im not sure 10k is big enough) Or is that what you are
doing?

(Smoke Eater/55383) What is the database? Oracle? SQL Server? MySQL? PostgreSQL?

Smoke Eater@rss.iscabbs.com — Tue, 29 Nov 2011 01:23:00 -0600

What is the database?  Oracle?  SQL Server?  MySQL?  PostgreSQL?

And I strongly second BDavis' recommendation.  Add a server for the query
database and ramp it up in capabilities, and have replication between the OLTP
database and the query database.

Beef up the network connection between the query database and the OLTP database
if you're worried about performance, so that things get replicated almost
immeidately.

(BDavis/55382) I would add moving the query database off the production OLTP se...

BDavis@rss.iscabbs.com — Mon, 28 Nov 2011 19:12:00 -0600

I would add moving the query database off the production OLTP server. That
would be a 'best practice' for a data warehouse. Depending on your user base
you may even want to consider reporting tools built for data warehouses, rather
than a 'services' layer, but that could be overkill and it just depends.

Otherwise seems like a workable plan.

(Smoke Eater/55381) While John Public's solution might be technically sound, you hav...

Smoke Eater@rss.iscabbs.com — Mon, 28 Nov 2011 16:12:00 -0600

While John Public's solution might be technically sound, you have to allow for
the politics involved.  I think your plan is sound, JL, and you can implement
parts of it concurrently.  Starting with upgrades to the DB hardware is fine,
but eventually you're going to need a data warehouse that queries hit... 
Architect that right and you'll be fine.

(The After Party/55380) JL

The After Party@rss.iscabbs.com — Mon, 28 Nov 2011 11:36:00 -0600

JL

  I'm not a DBA, but I think your 'solution' is fairly sound. If there is data
  that doesn't change "frequently", then you can save CPU cycles and real
  user time by aggregating it ahead of time.
  My first job, we had denormalized tables (nick named 'the twin towers' cuz
  we had the real data and the 'queued' version...people said the lights
  flickered when we did CRUD on them, heh) to be fast (and it was, though
  I never did a normalized version to compare).
  Do you have a (forgets DBA term) explain(?) plan(?) that shows your
  sum() is the 'bottleneck' or at least using a lot of time?
  Ask the DBAs to show you what queries are slow or which queries use
  the most CPU in a given 24 hour period.

(John Public/55379) shrug Then put a trigger on updates and see if that fixes it.

John Public@rss.iscabbs.com — Mon, 28 Nov 2011 11:27:00 -0600

*shrug* Then put a trigger on updates and see if that fixes it.

(JL/55378) TAP> What I mean is, I don't want to re-aggregate over and over ...

JL@rss.iscabbs.com — Mon, 28 Nov 2011 10:48:00 -0600

TAP> What I mean is, I don't want to re-aggregate over and over while the data
never changes.

While the data will change often, it will do so for individual entities, time
states, etc. It's not like we have to re-aggregate everything. So some
pre-computed data may stay valid for days or months.

> John Public
> That's what happens without a middle tier.

Oy, not this again. You want to change the problem, so it's more easily
solvable. Great, I get that. I'd like to also. But the fact remains that there
are hundreds of people interacting directly with the database to run queries,
and inserting a middle tier through which everyone must go is just unrealistic.

My strategic plan is to:
 - Fix the performance problems by
   - Boosting db hardware
   - Pre-computing and storing aggregate data (warehouse)
   - Re-writing slow queries
 - Do an audit of all direct access to the db
 - Create individual user id's, permissions, and views to satisfy those needs
 - Slowly build up a "services" layer through which all data will be retrieved
 - Follow that up with a services layer for all CRUD operations
 - Migrate everything to the service
 - Turn off direct DB access

I'm not sure it's possible. If it's possible at all, it's probably a 5-year
timeframe. But I'm starting with #1.

(John Public/55377) That's what happens without a middle tier. Everyone scratches th...

John Public@rss.iscabbs.com — Mon, 28 Nov 2011 08:12:00 -0600

That's what happens without a middle tier. Everyone scratches their head trying
to figgerout the least suckiest way to cram app logic in the db or data
optimizations in the app, and since their are no architects on the team, the
light bulb goes on...

(The After Party/55376) JL

The After Party@rss.iscabbs.com — Mon, 28 Nov 2011 07:42:00 -0600

JL

"So I'm just looking for a way to avoid aggregating data over and over"


"I need everything to be done real-time, so there is no lag at all (other than
perhaps some delay on insert/update while things are aggregated).

Furthermore, I need to join tables of static data with the aggregated data, so
a separate database doesn't really work."

  Seems to be you are contradicting yourself.
  Yeah, you want to avoid aggregating data over and over, but the data
  changes frequently, so you...have to re-aggregate it, no?
  There is no cake and eating it too.

  Almost feels like you're moving (right or wrong, I can't say) your
  problem you were trying to solve with object cache in memory to this
  aggregated db table. You'd then need to solve the problem of when is
  the (cache) aggreegated data stale? heh.

(BDavis/55375) A department at work recently had quotes for Goldengate, Attunit...

BDavis@rss.iscabbs.com — Wed, 23 Nov 2011 19:26:00 -0600

A department at work recently had quotes for Goldengate, Attunity, and
Infosphere. As I recall, they were in the $60k-$90k range. Pricing will vary a
lot though, and it is nearing the end of the year, so you might get it under
$50k. Or maybe some of the other vendors would be cheaper, I dunno.

(JL/55374) This is inside of an investment bank, so getting software approv...

JL@rss.iscabbs.com — Wed, 23 Nov 2011 17:08:00 -0600

This is inside of an investment bank, so getting software approval may be a bit
tricky, a cost really isn't an issue if it's the best solution.

Right now they have Business Objects hitting an Oracle data warehouse for a lot
of reports. But the main system hits the Sybase operational database where they
maintain entities and transactional data. The operational DB needs to be sped
up, and the main system often does sum() and other aggregate functions for
somewhat large transactional tables (3-5 million rows).

We can't point the main system to Oracle, nor can we re-write the whole main
system. So I'm just looking for a way to avoid aggregating data over and over
and over every day, and instead do it once on insert/update and modify the main
system by tweaking the queries.

I will look into the "BI World" for a product that would simplify the real-time
ETL approach, which could easily be used to copy data to other tables in the
same database, I presume.

Anything under $20k would probably be a no-brainer, but even $50k wouldn't be
out of the question.

(BDavis/55373) Keep in mind additional procedures and triggers are likely going...

BDavis@rss.iscabbs.com — Wed, 23 Nov 2011 16:23:00 -0600

Keep in mind additional procedures and triggers are likely going to slow things
down in a transactional system. Also materialized views have to be refreshed.
The normal approach would be a second database, with either realtime streaming
ETL or some sort of change data capture. These are hot products in the BI world
and there are lots of products, from database vendors as well as 3rd parties.
But it won't be cheap, so I hope you have a decent budget.

(Smoke Eater/55372) What kind of static data?

Smoke Eater@rss.iscabbs.com — Wed, 23 Nov 2011 14:59:00 -0600

What kind of static data?

(JL/55371) In fact, that already exists. The problem is, the ETL's are not ...

JL@rss.iscabbs.com — Wed, 23 Nov 2011 14:46:00 -0600

In fact, that already exists. The problem is, the ETL's are not quite
real-time. They are up to 15 minutes behind the operational database.

I need everything to be done real-time, so there is no lag at all (other than
perhaps some delay on insert/update while things are aggregated).

Furthermore, I need to join tables of static data with the aggregated data, so
a separate database doesn't really work.

(Smoke Eater/55370) Yes... The Data Warehouse Toolkit is the title of a good book t...

Smoke Eater@rss.iscabbs.com — Wed, 23 Nov 2011 14:30:00 -0600

Yes...  The Data Warehouse Toolkit is the title of a good book to read.

What I would suggest is to have a second database, and on updates to the first
database, have an ETL process to get the data in the second database, organized
like a data warehouse, so that you can do your reads from there.  That will
increase performance a great deal.

(JL/55369) I'm beginning a project to optimize a 300-table Sybase database ...

JL@rss.iscabbs.com — Wed, 23 Nov 2011 10:46:00 -0600


I'm beginning a project to optimize a 300-table Sybase database to increase
performance.

The current data is very normalized, transactional data. If I want to know how
many widgets we had last month, for example, I need to sum up all the entries
in the widget register that are <= that date.

Since reads outnumber writes by about 100 to 1, this is not optimal. One way
I'd like to increase performance is to aggregate and roll up data at the time
of update, rather than at the time of query.

I'm no database guru, so I'm looking for help in knowing the best way to
approach this. Some random thoughts:
 - Triggers on insert/update that call a stored proc to update a second
   denormalized table
 - Access everything via stored procs - no writing directly to tables allowed
 - Try to use something like materialized views or some other database-specific
   way that automatically aggregates data for me, so I can write my queries
   as I always would.

Any suggestions on what to read and how to learn more about this?

(Egregious/55368) Yeah, this is for a replicated database. The replicated databas...

Egregious@rss.iscabbs.com — Tue, 22 Nov 2011 07:50:00 -0600

Yeah, this is for a replicated database.  The replicated database
is living on a machine in the cloud and if it gets broken into we
don't want them to get back into the source machine.

I took out the "host all" lines in pg_hba.conf, but left in "host
replication."  The replicated machine can still get in via a
forwarded port, but I don't think anything else can connect over
that port.

Normal processes on that box can still connect to the Unix socket
and be fully trusted.

(Danix/55367) Egregious> Are you actually using different users? Try

Danix@rss.iscabbs.com — Tue, 22 Nov 2011 07:42:00 -0600

Egregious> Are you actually using different users? Try
psql -U mahuser mahdb
vs
psql -h 127.0.0.1 -U mahuser mahdb

also, check out your pg_hba.conf file, there might be differences for IP vs.
UNIX socket stuff.

(Raubvogel/55366) On MSSQL 2008, how do I set the debug level and where will it be

Raubvogel@rss.iscabbs.com — Tue, 22 Nov 2011 04:57:00 -0600

        On MSSQL 2008, how do I set the debug level and where will it be
written to?

(Egregious/55365) In Postgres, how can I have connections coming into the TCP sock...

Egregious@rss.iscabbs.com — Mon, 21 Nov 2011 12:45:00 -0600


In Postgres, how can I have connections coming into the TCP socket
only have read-access, while people connecting via the Unix socket
have full access.

This is for essentially a single-user platform, so I can mess with
"users" however I want.

(John Public/55364) Fwiw, I did measure the various approaches yesterday. The most ...

John Public@rss.iscabbs.com — Mon, 21 Nov 2011 08:46:00 -0600

Fwiw, I did measure  the various approaches yesterday. The most important
factor is whether the where clause is on a KEY or INDEX, and the result set is
small. In that case, the actual query execution times are nearly identical,
but the simple approach is faster because there is no latency between the two
calls.

The "LIMIT 10 OFFSET R" approach requires one scan to get the count, followed
by a scan to offset R, then whatever to fetch the rows. The scan to a random
offset is costly. Its roughly O(N + L + M/2). N is the total rows, and M is
restricted result set. M/2 is the expected random offset. L is the latency
between the two queries.

The "ORDER BY RAND() LIMIT 10" approach requires a scan to get the rows, and a
simultaneous top-n sort to order them. The sort is not costly only when the
result set is small. It's roughly O(N + M*log(M)).

If the result set is large, then both approaches are actually TERRIBLE in
practical terms. But Faunus's approach scales better in that case.

(Egregious/55363) LIMIT 1 and LIMIT 10 will probably be about the same, at least i...

Egregious@rss.iscabbs.com — Mon, 21 Nov 2011 06:53:00 -0600

LIMIT 1 and LIMIT 10 will probably be about the same, at least in Postgres.

Postgres does this:

         Sort Method:  top-N heapsort  Memory: 165kB

when I give it a LIMIT of about 1080 or less.  (At least on this 1 table
I'm testing against.)

Finding the biggest M elements in a set (where M is a constant) is
linear time.  And although databases frequently disappoint me in not
knowing the best way to do things, this is one where they seem to
Do The Right Thing.

(Faunus/55362) If I go ask it to ORDER BY RAND LIMIT 10 it's a lot faster than ...

Faunus@rss.iscabbs.com — Mon, 21 Nov 2011 06:50:00 -0600

If I go ask it to ORDER BY RAND LIMIT 10 it's a lot faster than asking it to
ORDER BY RAND LIMIT 1 ten times, but that might be more about the number of
trips to the DB from PHP than about anything the DB is doing.

(Egregious/55361) I'm pretty sure databases are smart enough to only keep 1 row wh...

Egregious@rss.iscabbs.com — Mon, 21 Nov 2011 06:19:00 -0600


I'm pretty sure databases are smart enough to only keep 1 row when you
say "ORDER BY RAND() LIMIT 1".   I just tested in PostgreSQL, which
I've been using much more than MySQL these days, and it's a lot faster
to keep just 1 result.

(Faunus/55360) OK, I went to ORDER BY RAND().

Faunus@rss.iscabbs.com — Sun, 20 Nov 2011 08:17:00 -0600

OK, I went to ORDER BY RAND().

performance was not great compared to my sketchy previous solution, but I can
get around that by always grabbing a bunch of them at once if I'll need more
than one, rather than making multiple calls.  Like if I need 10 random names,
have it give me a LIMIT 10 instead of LIMIT 1.

That means I don't get the possibility of repetition if I want 10 random names,
but that's fine.

(John Public/55359) The offset only makes sense if there is an order to the set in t...

John Public@rss.iscabbs.com — Sun, 20 Nov 2011 05:42:00 -0600

The offset only makes sense if there is an order to the set in the first
place. So basically, you can't escape the ORDER BY. Your approach could be
faster *if* you ORDER BY a primary key and take the randomized OFFSET (as
opposed to ORDER BY RAND()) only because certain types of primary keys are
already ordered, so I'd hope that there would be no "shuffling" involved.

Also, you've got a race condition unless those two PHP calls are in the same
transaction.

(Faunus/55358) It was more like this, and I forgot -- I actually needed to grab...

Faunus@rss.iscabbs.com — Sat, 19 Nov 2011 14:46:00 -0600


It was more like this, and I forgot -- I actually needed to grab a
random item *of a particular type* from the table, so it was a little
more complicated than I mentioned at first.

$stmt = $the_pdo->prepare('SELECT COUNT(*) from `names` where `type` = ?');
$stmt->execute(array($type));
$row = $stmt->fetch();
$count = $row[0];
$randomized = rand(0, $count-1);
$stmt = $the_pdo->prepare('SELECT `name` from `names` where `type` =
                            ? LIMIT 1 OFFSET '.$randomized);
$stmt->execute(array($type));
$row = $stmt->fetch();
return $row[0];

(John Public/55357) Faunus> There are many ways. None are very satisfying. Choose yo...

John Public@rss.iscabbs.com — Sat, 19 Nov 2011 09:53:00 -0600

Faunus> There are many ways. None are very satisfying. Choose your poison.

1.  SELECT * from T order by RAND() LIMIT 1;

    Advantage: Correct result. Simple code.
    Problem:   Performance. Must order entire result set.

2.  SELECT * from T where T.PK >=
        (SELECT FLOOR(MIN(PK) + RAND()*(MAX(PK)-MIN(PK))) FROM T)
        ORDER BY PK LIMIT 1;

    Advantage: Single join.
    Problem:   Incorrect results.

3.  select * from T where T.PK = (SELECT PK from T ORDER BY RAND() LIMIT 1);

    Advantage: Correct result. Ordering is performed on key set.
    Problem:   Performance. Must order entire KEY set.

4.  start transaction;
    select @tot := COUNT(*),
           @min := MIN(PK),
           @max := MAX(PK) from T;
    select * from T where pk >= FLOOR(@min + RAND()*(@max-@min)) LIMIT 1;
    commit;

    Advantage: Just a readable variation of #2.
    Problem:   Still incorrect results.

5.  $total = $php_db_call ("select count(*) from T");
    $id = rand ($total);
    $row = $php_db_call ("select * from T where pk = $id")

    Advantage: Makes perfect sense to PHP coders.
    Problem: Race condition. Probably will not even return any rows.


In summary, benchmark #1, and if it sucks, use #3.  Don't use the others
despite what your favorite LAMP blog recommends.  If this is critical, and
must be performed repeatedly, consider generating a separate, randomly ordered
table, with a strict sequence, and walking through that sequence each time you
need a new row. Like so:

    /* TABLE O is a random order of table T */
    create table O (
        seq int not null auto_increment primary key,
        tpk int not null,
        foreign key TPK references T (PK) ON DELETE CASCADE
    ) ENGINE=InnoDB;
    INSERT INTO O SELECT NULL, PK from T ORDER BY RAND();

    /* Fetch a random row from T using O */
    select * from T join O on T.pk=O.tpk
        where o.seq = prev_seq;
    ++prev_seq;

(The After Party/55356) Faunus

The After Party@rss.iscabbs.com — Sat, 19 Nov 2011 09:56:00 -0600

Faunus

  Hrm, good point. I didn't htink of it that way. Maybe it depends on
  the index? Bench it for us!

(Faunus/55355) Well, my (possibly flawed) understanding is that in order by RAN...

Faunus@rss.iscabbs.com — Sat, 19 Nov 2011 09:54:00 -0600

Well, my (possibly flawed) understanding is that in order by RAND you would end
up sorting the whole table in order to grab one line.

In my version, I guess, you end up *counting* the whole table and then grabbing
one line from an offset...  intellectually it seems much more straightforward,
but I don't know, which is why I asked.

(The After Party/55354) Faunus

The After Party@rss.iscabbs.com — Sat, 19 Nov 2011 09:51:00 -0600

Faunus

  Why is order by rand() offset limit 1 any sillier than what you're doing?
  Yours sounds like more work...

(Faunus/55353) I need to select a random row from a 5000+ row table. I'm just ...

Faunus@rss.iscabbs.com — Sat, 19 Nov 2011 09:38:00 -0600


I need to select a random row from a 5000+ row table.  I'm just doing a COUNT()
to get the number of rows, picking a random number, and then selecting that row
as an OFFSET LIMIT 1.  Is that a good way to go or is there some clever SQL fu
way to go?  This is MySQL.  I've found people talking about doing an ORDER BY
RAND() but that seems silly to me on the face of it.

(Faunus/55352) HA! had to quote `lines`. Must be a keyword. Should find a bet...

Faunus@rss.iscabbs.com — Thu, 17 Nov 2011 15:59:00 -0600

HA! had to quote `lines`.  Must be a keyword.  Should find a better name for
that column.

(Faunus/55351) GAH

Faunus@rss.iscabbs.com — Thu, 17 Nov 2011 15:13:00 -0600


GAH

WHYMESTUPID

What the hell am I doing wrong to generate this:

Fatal error: Uncaught exception 'PDOException' with message
'SQLSTATE[42000]: Syntax error or access violation: 1064 You have an
error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near 'lines =
'prefix1\nprefix2\nprefix3' WHERE id = '1'' at line 1' in (blah blah
blah my php file)

Is it that id should be an int, not a string?

I'm letting php's pdo construct this query for me.

It looks like this:

$stmt2 = $pdo->prepare(
'UPDATE spinfile_extras SET lines = :lines WHERE id = :id');

$stmt2->execute(array(':lines' => $lines, ':id' => $intid));

where $intid is the number 1, and $lines is
"prefix1\nprefix2\nprefix3".

THere's got to be something very simple I'm just not seeing here.

Help?

(Faunus/55350) That's the stuff. I'll try some of that. Thanks. :)

Faunus@rss.iscabbs.com — Tue, 15 Nov 2011 12:25:00 -0600

That's the stuff.  I'll try some of that.  Thanks. :)

(John Public/55349) select group_id, count(*) from item group by group_id.

John Public@rss.iscabbs.com — Tue, 15 Nov 2011 11:29:00 -0600

select group_id, count(*) from item group by group_id.

select group.id, group.name,  ETC.., COUNT(*)
       count(*) from group join item on group.id=item.group_id
       GROUP BY  select group.id, group.name,  ETC..

(Faunus/55348) Help my feeble SQL.

Faunus@rss.iscabbs.com — Tue, 15 Nov 2011 11:08:00 -0600

Help my feeble SQL.

I have a table of GROUPS and a table of ITEMS.  each ITEM belongs to a
GROUP (that is, it has a group_id that corresponds to a group's id).

I want a list of GROUPS and with each GROUP I want a count of the ITEMs
associated with it.

I know I want to do some joining, grouping, and counting, but it's going to
take me some beating my head against docs to find out, and I bet
somebody here could spit out the answer off the top of their head the
way I wish I could.

(Smoke Eater/55347) I have a continuous form. It shows a row of records. One field...

Smoke Eater@rss.iscabbs.com — Mon, 14 Nov 2011 02:08:00 -0600

I have a continuous form.  It shows a row of records.  One field is a checkbox.
I want to be able to have the next field visible, active, and enabled only if
the checkbox is checked.  If it's unchecked, I want it to disappear.

I can make this happen, but it happens for all rows at once.  I want it to
happen only for one row, the row where I check the checkbox.

I've been doing searching, but it seems like that's not possible for continuous
forms.

(BDavis/55346) I used to use Access a lot, but not so much in last version or t...

BDavis@rss.iscabbs.com — Sun, 13 Nov 2011 18:53:00 -0600

I used to use Access a lot, but not so much in last version or two. Maybe you
could post a synopsis of the issue and we can tell you if we have a clue?

(Danix/55345) I do know a ladyfriend who could be considered an Access guru. S...

Danix@rss.iscabbs.com — Sun, 13 Nov 2011 17:54:00 -0600

I do know a ladyfriend who could be considered an Access guru. Sadly, I have
lost communication with her. :(

(Smoke Eater/55344) Is there anyone reading this who considers themselves an Access ...

Smoke Eater@rss.iscabbs.com — Sun, 13 Nov 2011 13:26:00 -0600

Is there anyone reading this who considers themselves an Access guru?  I'm
having a very obscure problem with Access and need help.

(John Public/55343) Cannot and should not what?

John Public@rss.iscabbs.com — Wed, 02 Nov 2011 05:38:00 -0500

Cannot and should not what?
AUTO_INCREMENT starts at 1.
You can set it to whatever you want via ALTER.
You should set it if you need it to start at something other than 1.
It will be reset to its starting value on a TRUNCATE.

(River Rat/55342) In short, you cannot and should not. In MySQL, if you have a col...

River Rat@rss.iscabbs.com — Wed, 02 Nov 2011 05:25:00 -0500

In short, you cannot and should not. In MySQL, if you have a column that is
assigned the AUTO_INCREMENT count, you can reset that count by using this
statement (or check out the Operations tab in phpMyAdmin):

ALTER TABLE tablename AUTO_INCREMENT=0

(River Rat/55341) I believe it is zero when it starts out. But turns into when as...

River Rat@rss.iscabbs.com — Wed, 02 Nov 2011 05:24:00 -0500

I believe it is zero when it starts out.  But turns into when as you are
entering the first record?

(John Public/55340) TRUNCATE TABLE t;

John Public@rss.iscabbs.com — Wed, 02 Nov 2011 04:18:00 -0500

TRUNCATE TABLE t;
or,
ALTER TABLE t AUTO_INCREMENT=1;  /* default is 1, not 0. */

Notes: InnoDB won't let you change the sequence to be less than the current max
value for that column. MyISAM will. TRUNCATE deletes all the rows, ALTER does
not.

(River Rat/55339) Is there any way to zero out the index value on MySql.

River Rat@rss.iscabbs.com — Wed, 02 Nov 2011 00:56:00 -0500

Is there any way to zero out the index value on MySql.

I have this table set up.

id
first name
last name
street address
city
state
zip

I added some records for testing purposes.
Then deleted them.
I would like to zero the id field like it was a clean database.
There isn't any data in the database.  Just would like to start
with an 1 in the id field.

(John Public/55338) It's the size of the largest single BLOB (or TEXT). I don't thin...

John Public@rss.iscabbs.com — Tue, 01 Nov 2011 03:37:00 -0500

It's the size of the largest single BLOB (or TEXT). I don't think setting it
in mysqldump will do anything. Here's why. According to the docs, the only
reasons you should hit that limit are:

    1. You have a single LONGBLOB (or LONGTEXT) larger than 1 GB. This case is
       unsolvable, AFAIU, since 1 GB is MySQL's communication protocol limit.

    2. The server's `max_allowed_packet` setting is too low. This is most
       likely the case in a normal setup.

    3. The client's `max_allowed_packet` setting is artificially low. This is
       not likely the problem in a normal setup, since the client default is
       1GB. Someone (a DBA, or distro packager) would have had to deliberately
       deflate that setting, which would serve no purpose.

The upshot: Set the value to 1GB in the server's config, not the client's.
The documented reason for the max_allowed_packet limit is to detect bugs or
rogue packets that would blow memory. Seems like a well-intentioned but poorly
designed measure, IMO.

(Raubvogel/55337) Mysql question: If I am exporting a db using mysqldump and am ge...

Raubvogel@rss.iscabbs.com — Tue, 01 Nov 2011 02:17:00 -0500

        Mysql question: If I am exporting a db using mysqldump and am getting
the packet too large message, how to find out the right value for
--max_allowed_packet?

(BDavis/55336) I'm not the ESRI guy but I will ask him about that, thanks. Righ...

BDavis@rss.iscabbs.com — Tue, 18 Oct 2011 14:32:00 -0500

I'm not the ESRI guy but I will ask him about that, thanks. Right now he is
sitting on Oracle but has done some testing with PostgreSQL. I honestly don't
know if he is making use of the Oracle native types or not. As far as I know
there is not a lot of editing and more serving up content, but he is very
knowledgeable and very likely to have looked into it at some point.

(Marigolds/55335) PostGIS is by far the dominant FOSS4G database option. On the pa...

Marigolds@rss.iscabbs.com — Tue, 18 Oct 2011 05:46:00 -0500

PostGIS is by far the dominant FOSS4G database option. On the pay side, it is
ArcSDE on top of sql server or oracle with spatial objects.
Make sure to use Geometry with projected data and geography with unprojected
data.
The other various FOSS4G database options are underdeveloped while the pay
options other than ESRI are sometimes better, but without the market share and
3rd party developers.
If you are using ESRI, I highly recommend switching to using the native spatial
types in the underlying database(though the ESRI file geodatabase is often the
fastest option if you are not using versioned editing and long transactions).
The bleeding edge for GIS right now though is NoSQl type options such as fusion
tables with earth builder, tilemill, and tiled vectors.

(Darkhaven/55334) uh, last time i looked, geospacial fields in mysql were not in s...

Darkhaven@rss.iscabbs.com — Tue, 18 Oct 2011 04:52:00 -0500

uh, last time i looked, geospacial fields in mysql were not in stock either -
they were plugins or some such and considered slow. it's why i convert my dms
to lat/long. since opengis was mentioned in here, i might as well mention
perl's PDL stuff which is quite nice for this type of thing and pretty fast
(though i really don't have another source for comparison on the speed :) )

(BDavis/55333) MSSQL and Oracle also support geospatial, at least in current ve...

BDavis@rss.iscabbs.com — Mon, 17 Oct 2011 11:16:00 -0500

MSSQL and Oracle also support geospatial, at least in current versions.
PostgreSQL has a plug-in for it called PostGIS. I know PostgreSQL is supported
by quite a few GIS vendors, not sure if they use PostGIS or not.

Geospatial and data mapping is one of those hot topics in certain segments of
IT at the moment. I have seen some neat demos and we are just starting to apply
it in my work group (BI), in fact it is part of a current project. But we have
an ESRI server so we are not really using any database features directly as our
data is already geocoded within relevant tables.

(Faunus/55332) Whoa! That is cool.

Faunus@rss.iscabbs.com — Mon, 17 Oct 2011 08:55:00 -0500

Whoa!  That *is* cool.

(John Public/55331) Faunus> OpenGIS. It's pretty nifty. All it really is is a geomet...

John Public@rss.iscabbs.com — Mon, 17 Oct 2011 08:50:00 -0500

Faunus> OpenGIS. It's pretty nifty. All it really is is a geometry library
linked into the DB. They've been around forever, but suddenly became useful for
map searches ("restaurants near Topeka").
http://www.opengeospatial.org/
For MySQL, it means you get built-in spatial data types and operations on them.

(Danix/55330) CSV> The poor man's EXPORT option?

Danix@rss.iscabbs.com — Mon, 17 Oct 2011 07:28:00 -0500

CSV> The poor man's EXPORT option?

(Faunus/55329) "Geospatial data"??

Faunus@rss.iscabbs.com — Mon, 17 Oct 2011 07:21:00 -0500

"Geospatial data"??

(John Public/55328) Most of the engines are specialized for specific purposes and do...

John Public@rss.iscabbs.com — Mon, 17 Oct 2011 06:32:00 -0500

Most of the engines are specialized for specific purposes and do not really
"compete" with one another.

GENERAL ENGINES:

    MyISAM - Benefits: Speed. Delayed writes. Filesystem-level snapshots.
             Encryption, compression, replication, geospatial data.
             Drawbacks: No transactions. No FK constraint checks. Table-level
             locking.

    InnoDB - Benefits: Transactions. Row-level locking. FKs. Consistent reads.
             Encryption, compression, replication, geospatial data.
             Drawbacks: Slower, especially for batch inserts/updates or DB
             restores. Constraint checks are not delayed.

SPECIALIZED ENGINES:

    Memory - Benefits: Exists only in RAM, so is hyperfast.
             Drawbacks: Not stored. No transactions. Many features do not work
             on memory tables.

    Merge  - Benefits: A much simpler, faster form of partitioning.
             Drawbacks:  Only works on MyISAM tables. Kinda weird.

    Federated - These are what Microsoft would call "Linked Tables".
             Each table can exist on a remote machine. It's just a broker
             between two tables. It operates on the TABLE level.

    Archive - Compressed tables. No indexes.  MySQL now supports compression
            directly, so this engine is becoming obsolete.

    Blackhole - Storage is a no-op. Apparently invented for testing and
                troubleshooting but they've since discovered cute tricks to
                use it for.

    CSV - Stores data in comma-separated files.  I have no idea WTF this is
          good for.

(Faunus/55327) So what are the benefits/drawbacks of the various MySQL storage ...

Faunus@rss.iscabbs.com — Mon, 17 Oct 2011 06:03:00 -0500

So what are the benefits/drawbacks of the various MySQL storage engines?

(John Public/55326) MyISAM is still the default. I guess I don't see it as confusin...

John Public@rss.iscabbs.com — Mon, 17 Oct 2011 05:50:00 -0500

MyISAM is still the default.  I guess I don't see it as confusing.  It's
verbosely documented in MySQL that the DBA chooses which storage engine he
wants. This is no different than other systems where you have to choose the
internal index mechanism for INDEXES, or specify the storage layout. It seems
to me like something the damn DB oughta "just do".

(Danix/55325) JP> When I did the big switch, MyISAM was still the default. In ...

Danix@rss.iscabbs.com — Mon, 17 Oct 2011 05:47:00 -0500

JP> When I did the big switch, MyISAM was still the default. In fact, one
production database got fscked because the original dev had thought he had
created InnoDB tables, that must've been sometime around 2006. In fact, that
particular MySQL install had innodb disabled as a default setting. Why, oh why?

It seems the good fixes came after I stopped using MySQL, though.

(John Public/55324) Doesn't it have circular dependency problems or something? I rec...

John Public@rss.iscabbs.com — Sun, 16 Oct 2011 19:06:00 -0500

Doesn't it have circular dependency problems or something? I recall one of the
DBAs telling me this recently. He saw some of my MySQL schema code and told me
that SQL Server wants all the constraints in separate statements AFTER the
CREATE TABLE statements. As in,

        create table foo (....);
        create table bar (....);

        create constraint foreign key foo(bar_id) references bar(id);
        create constraint foreign key bar(foo_id) references foo(id);

(BDavis/55323) Been there since at least SQL2000.

BDavis@rss.iscabbs.com — Sun, 16 Oct 2011 19:01:00 -0500

Been there since at least SQL2000.
http://msdn.microsoft.com/en-us/library/aa258255(v=sql.80).aspx

(John Public/55322) Boogie> The part that says it matters where the FK definition is...

John Public@rss.iscabbs.com — Sun, 16 Oct 2011 18:39:00 -0500

Boogie> The part that says it matters where the FK definition is found in the
CREATE TABLE statement. If the FK is defined in the same clause as the column
itself, then InnoDB ignores it. If the FK is defined in its own clause, then
the column constraint is applied.

I thought about this some more. And, while it sucks completely, it is not
likely to bite *me*, because for /years/ InnoDB's FK constraints only worked
if you first defined an INDEX on the column. So the only way to actually
create a FK was in a separate clause, after the clause that created the INDEX.
i.e., you never coded a CREATE TABLE with FK clauses inline with the columns.

fwiw, I *think* SQL Server still doesn't allow inline FK/PK clauses.

Danix> Eh, that's more of a myth about MySQL than reality. In practice,
everyone knows that InnoDB is the only engine that supports standard SQL
constraints. So it's not like in practice somebody would create a MyISAM
table, then wonder why the FKs aren't working.  (Don't get me wrong, there are
plenty of gotchas and flaws in MySQL, but confusing the table engines isn't
one of them. They are pretty obvious: InnoDB is used for real projects. MEMORY
is used for transient data.  And MyISAM is used by PHP CMS's!

(BDavis/55321) That and an unknown quantity of other "gotchas" is why I never e...

BDavis@rss.iscabbs.com — Sat, 15 Oct 2011 18:07:00 -0500

That and an unknown quantity of other "gotchas" is why I never even started
using MySQL for anything other than a few non-transactional web sites. It
is actually a lot more full-featured than it used to be. Still, I just don't
trust that I will get results I have come to expect from every other database I
use on a regular basis.

(Danix/55320) See, that's one of the reasons I switched back to Postgres. I go...

Danix@rss.iscabbs.com — Sat, 15 Oct 2011 16:46:00 -0500

See, that's one of the reasons I switched back to Postgres. I got these "gems"
to bite me:

"For other storage engines, MySQL Server parses and ignores foreign key
specifications."

You might start a transaction, do a crapton of things and then need a ROLLBACK,
getting a success on that. Except it isn't, because one of the tables uses the
transaction-less MyISAM. Same with creating Foreign Keys .... you don't get a
warning if you don't use InnoDB as an engine.

(Boogie With Stu/55319) Which part made you spit your coffee? The fact that FKs are onl...

Boogie With Stu@rss.iscabbs.com — Sat, 15 Oct 2011 16:33:00 -0500

Which part made you spit your coffee?  The fact that FKs are only possible on
the InnoDB engine, or that the "inline REFERENCES specifications" are ignored
despite the SQL standard?

(John Public/55318) I've been using MySQL since the late 90's. I'd call myself an "e...

John Public@rss.iscabbs.com — Thu, 13 Oct 2011 09:14:00 -0500

I've been using MySQL since the late 90's. I'd call myself an "expert" in
MySQL, whatever that means. But I just read this little note, buried pages
deep in MySQL's `CREATE TABLE` docs. I nearly spit out my coffee...

    Furthermore, InnoDB does not recognize or support "inline REFERENCES
    specifications" (as defined in the SQL standard) where the references are
    defined as part of the column specification. InnoDB accepts REFERENCES
    clauses only when specified as part of a separate FOREIGN KEY
    specification. For other storage engines, MySQL Server parses and ignores
    foreign key specifications.

    Source: http://dev.mysql.com/doc/refman/5.5/en/create-table.html

(Cyanide/55317) I dont know but Oracle Forms compatibility is on the top ten for...

Cyanide@rss.iscabbs.com — Wed, 12 Oct 2011 07:20:00 -0500

I dont know but Oracle Forms compatibility is on the top ten for the FSF list.
They -may- have a project started already that needs help.

(Darkhaven/55316) any recommendations for an oracle 'studio' besides their java th...

Darkhaven@rss.iscabbs.com — Tue, 11 Oct 2011 13:26:00 -0500

any recommendations for an oracle 'studio' besides their java thing and toad
(which is $870 or $2k+ ) ? i'd prefer something open source but really just
wondering what the options are as well.

(Danix/55315) The SQL Server installer is supposed to open up port 1433. Exc...

Danix@rss.iscabbs.com — Mon, 10 Oct 2011 13:38:00 -0500

The SQL Server installer is *supposed* to open up port 1433. Except when it
doesn't, then you have to do it manually.

(Smoke Eater/55314) No, opening port 1433 is the way to do it.

Smoke Eater@rss.iscabbs.com — Mon, 10 Oct 2011 12:10:00 -0500

No, opening port 1433 is *the* way to do it.

(Raubvogel/55313) BDavis, you are right. I ended up just opening port 1433 in the

Raubvogel@rss.iscabbs.com — Mon, 10 Oct 2011 11:54:00 -0500

        BDavis, you are right. I ended up just opening port 1433 in the
firewall. There is probably a more elgant way to do it, but that did the trick

(Danix/55312) netstat -ba should give you info on listening ports as well, and...

Danix@rss.iscabbs.com — Mon, 10 Oct 2011 10:47:00 -0500

netstat -ba should give you info on listening ports as well, and the -b option
means that the process listening/using the socket will be printed as well.

(BDavis/55311) It sounds like maybe the Windows Firewall is blocking it. Try di...

BDavis@rss.iscabbs.com — Mon, 10 Oct 2011 10:44:00 -0500

It sounds like maybe the Windows Firewall is blocking it. Try disabling the
firewall. If it works, then you'll need to add a rule to the firewall. If
not... not sure. Yes netstat exists on Windows, at a cmd prompt do a "netstat
-?" for syntax help which as I recall is a bit different from say Linux.

(Raubvogel/55309) Easy sqlserver2008 in Windows 7 question: trying to enable tcp

Raubvogel@rss.iscabbs.com — Mon, 10 Oct 2011 10:31:00 -0500

        Easy sqlserver2008 in Windows 7 question: trying to enable tcp
on it. Based on http://msdn.microsoft.com/en-us/library/ms165 718.aspx I
went to Server config Manager->Server network config->protocols for
MSSQLSERVER and enabled tcp/IP. And then went to tcp/ip properties and
enabled ip2, providing the host's ip address before restarting server.
But I still cannot connect to the server; I am getting a

SQLServer is unavailable or does not exist (20009)

message when trying to connect to it also, nmap tells the port 1433 is
filtered, not open, and telnet to the host port 1433 results in
nothing; I expected it to actually connect "escape character=" instead
of just haging at

telnet delta 1433
Trying 192.168.11.146...

Which means it is not connecting. What could I be missing here? Can I
netstat in windows 7?

(Darphbobo/55308) Thanks BDavis. We are doing a lot of different ETL processes so ...

Darphbobo@rss.iscabbs.com — Mon, 10 Oct 2011 06:19:00 -0500

Thanks BDavis. We are doing a lot of different ETL processes so I am trying to
find a launchpad to start from. This looks like a good program thanks for the
help!!!

(BDavis/55307) You might find the Talend products interesting. There are lots o...

BDavis@rss.iscabbs.com — Thu, 06 Oct 2011 14:52:00 -0500

You might find the Talend products interesting. There are lots of tools for ETL
and data quality, but what they do varies a lot so it depends on what you need.
We are actually looking at getting something on the BI team I work on, as soon
as we find some time to do the research. :-/

(Darphbobo/55306) I am trying to do some automation with ETL testing. Are there an...

Darphbobo@rss.iscabbs.com — Thu, 06 Oct 2011 09:16:00 -0500

I am trying to do some automation with ETL testing. Are there any good
frameworks out there? I have looked at Scriptella and TSQL. Any good ones
specifically for incremental load and data validation?

(Stilgar/55303) Free SQL Formatter...pretty cool:

Stilgar@rss.iscabbs.com — Sat, 10 Sep 2011 03:47:00 -0500

Free SQL Formatter...pretty cool:
 http://poorsql.com/

Hmm, doesn't seem to be up right now, well here's a video of it in action...
 http://screencast.com/t/FX1aM1Ercg

(BDavis/55302) Depending on database, you can do something like

BDavis@rss.iscabbs.com — Tue, 06 Sep 2011 09:37:00 -0500

Depending on database, you can do something like
  insert into newtable
  ( select somecolumn, cast(anotherone as integer), cast(third as varchar(12)
    from oldtable )
then use whatever selects you feel are needed for QA between old and new.

(SilverEdge/55301) Would just changing numerics to integer cause a problem? Or var...

SilverEdge@rss.iscabbs.com — Tue, 06 Sep 2011 09:00:00 -0500

Would just changing numerics to integer cause a problem?  Or varchar to int?
obviously, I'd make a copy first, just in case.

Or should I just make a copy of each table for comparison, create duplicate
columns that are defined correctly, and copy them over using cast/convert.
then remove the original columns, and then compare the revised table to the
copy?

(SilverEdge/55300) Well, referring back to my post #55286, would this be the time t...

SilverEdge@rss.iscabbs.com — Tue, 06 Sep 2011 06:51:00 -0500

Well, referring back to my post #55286, would this be the time to recreate the
database correctly in SQL2005?  Also, what is the best way to back it up (again
referring to #55286)?  I feel like I'm doing what I should do, but I also don't
feel like it's working right.

(John Public/55299) Create a new table, or copy the old one (six and one half dozen ...

John Public@rss.iscabbs.com — Tue, 06 Sep 2011 06:19:00 -0500

Create a new table, or copy the old one (six and one half dozen of the other),
so you can compare the final table to the original after you have completed
the conversion. Since your conversion is lossless, you simply have to verify
that the outer join of the two tables is empty -- or that the inner join is
identity.

(SilverEdge/55298) It does sound complicated, but not undoable. Seems easier than ...

SilverEdge@rss.iscabbs.com — Tue, 06 Sep 2011 06:00:00 -0500

It does sound complicated, but not undoable.  Seems easier than creating a new
set of tables entirely.  Basically it's the same thing, just one less
complexity with the new table bit.

(BDavis/55297) I guess to me it would depend on how many columns in each table....

BDavis@rss.iscabbs.com — Tue, 06 Sep 2011 05:09:00 -0500

I guess to me it would depend on how many columns in each table. You could
create a new set of tables. That would be straightforward. But if it is only a
couple of columns in each table, you could create new columns, copy the data
into the new columns, drop the old columns, and rename the new with the old
name. That sounds more complicated than it is, I think, and if you have no
constraints it will be pretty simple.

Alternatively, you could download a trial of one of the many database tools and
let it do the heavy lifting. We use Sybase PowerDesigner, there are others.
Just be sure to take a backup first. :)

(SilverEdge/55296) I need advice on how to fix a database...

SilverEdge@rss.iscabbs.com — Tue, 06 Sep 2011 04:56:00 -0500

I need advice on how to fix a database...

So, I have a custom database that is being populated primarily by information
from another database.  When I originally made my custom one, I didn't have
much experience with them.  I created the fields and lengths based on what I
thought were reasonable, versus looking at what the other database used and
creating them similarly.  Accordingly, I have some fields that are varchar(100)
that only need to be varchar(30) and some that are varchar(10) that need to be
varchar(12).  I also have some that are numeric that should be integers.  The
custom database already has about 40,000 records in it, and we cannot lose this
data.  How would I go about fixing this?

I imagine, I cannot just go in and change the definitions.  Though, perhaps the
varchar(10) to varchar(12) I can do like that.  I imagine the numeric to int
isn't as simple?  What about a varchar changed into int?

A friend of mine said he recommended creating an entire set of new tables (I
was consistent with how I did it and so how it is messed up), and doing a
cast/convert to migrate the data into the new tables.

Another thing I'm lacking apparently is Foreign Keys, so that's not a concern
coming from the existing table, though, once things are fixed I plan on
specifying those.

What are your thoughts?  I'm looking to do this right.  The least time the
better of course, but it's more important to take the time to do it right.
Thanks in advance!

(Egregious/55295) I did it in OOffice, too. For all the complaints about the form...

Egregious@rss.iscabbs.com — Wed, 31 Aug 2011 07:33:00 -0500

I did it in OOffice, too.  For all the complaints about the format,
I could edit the diagram with tools already on my computer, which
ought to count for something.

(John Public/55294) I opened it in OpenOffice. It looks like a typical entity-attri...

John Public@rss.iscabbs.com — Wed, 31 Aug 2011 07:28:00 -0500

I opened it in OpenOffice.  It looks like a typical entity-attribute diagram
to me.  Is it a beautiful work of art? no, but I was able to make sense of it
in a blink.

(Egregious/55293) I was wondering if it was something generating in-class by a

Egregious@rss.iscabbs.com — Wed, 31 Aug 2011 06:55:00 -0500


I was wondering if it was something generating in-class by a
brainstorming session of the students.

Which lines are pictures of lines, BTW?  I tried about a dozen
and then wondered why I was looking so hard.

(John Public/55292) Oh, my bad. I must have been hammered about the head by the tim...

John Public@rss.iscabbs.com — Wed, 31 Aug 2011 06:32:00 -0500

Oh, my bad.  I must have been hammered about the head by the time I reached
the 4th scare quote in the very first sentence.  When some scare-quotes the
word "university", you know they're a dick regardless of what the post is
about! :)

I dunno. I'd have to know what the real story is behind that ERD, *then*
crucify the author! Heh.  Perhaps the ERD was intentionally vague about the
meaning. Perhaps it was auto-generated by something. Perhaps the school didn't
budget licensing for a real ERD tool.  Perhaps it was an exercise, and they
will refine the diagram as the course progresses.

Crack open a DB course textbook. It will strategically avoid notational-heavy
diagrams for most of the book. That's because they don't want to imply that
the relational model has anything to do with lines connecting boxes. They want
to distance the students from anything remotely resembling ERDs and vendor
implementations until the foundation is laid.  The first two-thirds of a DB
course will probably focus on concepts such as tuples, relations, domains,
attributes, functional dependencies, relational algebra, relational calculus,
but never mention those dirty little things called "foreign keys" until far,
far into the course, and even then only for lab homework.

...Besides even good ERDs look like a total mess. Those things are write-only!
;)

(Dr Doctor/55291) John Public -> You misread the article. The ERD was provided to...

Dr Doctor@rss.iscabbs.com — Wed, 31 Aug 2011 06:20:00 -0500


John Public -> You misread the article.  The ERD was provided to the class by
the professor.  A student passed it to the guy who posted the blog entry.  It's
not "less than professional homework" it's a "less than professional class hand
out."

(John Public/55289) What a dick. A student submits less-than-professional- quality ...

John Public@rss.iscabbs.com — Wed, 31 Aug 2011 05:18:00 -0500

What a dick.  A student submits less-than-professional- quality homework, and
this guy shits all over it. He suffers from an inflated sense of how important
ERDs are in the real world. (How many professionals do you know who bother to
create ERDs? How many students do you know who submit professional-quality
*anything*?)

(Stilgar/55288) This is pretty good:

Stilgar@rss.iscabbs.com — Wed, 31 Aug 2011 04:38:00 -0500

This is pretty good:
 http://weblogs.sqlteam.com/robv/archive/2011/08/30/acad-anemic.aspx

(Kraven/55287) Im having a problem with Crystal reports and exporting to a text...

Kraven@rss.iscabbs.com — Tue, 23 Aug 2011 09:47:00 -0500

Im having a problem with Crystal reports and exporting to a text file. Im
working on an export from a sql 2005 db to a fixed lenght file format. It was
working well but not when i export to txt the file is blank, if i export to rtf
and then save from there to txt it works but my users wont like those extra
steps. Any ideas whats gone wrong.