Unix Technical Forum

version/source control in datawarehouse project

This is a discussion on version/source control in datawarehouse project within the SQL Server Data Warehousing forums, part of the Microsoft SQL Server category; --> Hi! I'm a freshman in a datawarehouse company. All my coworkers are on-site in user side,such as banks, and ...


Go Back   Unix Technical Forum > Database Server Software > Microsoft SQL Server > SQL Server Data Warehousing

Register FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1 (permalink)  
Old 02-27-2008, 07:08 PM
YesGoGoGo
 
Posts: n/a
Default version/source control in datawarehouse project

Hi!
I'm a freshman in a datawarehouse company.
All my coworkers are on-site in user side,such as banks,
and our development machine is also given by user.
There is no server or desktop in our company.
After I join this company,I find there's no version/source control in
our company's projects.
Because I used to work in a web-based system development company,
I think it's VERY important to have code under version control.

In my new company,there are 1~3 members in each project.
Everyone of them will terminal to server,open the same project and
modify it.
I try to think if there is a better way to improve this develop cycle.
First of all,I have to have a version control server,such as
subversion.
Because we don't have a server in our company
,maybe installing SVN on development server is a good way.
When sb wants to modify the code(SSIS/SSRS),
he has to update his working copy to the latest version and modify it.

But for me,there is a problem
take SSIS for instance, how do we design datasource connectiion to
make it available for every developer?
if datasource is DB connection,I can change "localhost" to servername
or ip to ensure connections on everyone's computer works.
but for development server,would using servername instead of
"localhost" make db connection lower?because it has to parse the
servername? this is my first convern.

If datasource is file connection,there is a similar situation,using
relative path could make the code fit everyone's working environment
but maybe inefficient when reading file?

In the begining,I think maybe we can copy DB or files to developer's
machine, but I find DB and files are too large(too much data...)

And there are many abosolute path in our code
So if I want to bring version/source control into this project,it
seems we have to modify those path.

After ETL,we have SSAS,SSRS to do.
I want to know if there is a better way or best practice to design the
version control of the datawarehouse project?
For me,it seems like a mess now...

Need some advice
Thanks a lot!!

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #2 (permalink)  
Old 02-27-2008, 07:08 PM
Jeje
 
Posts: n/a
Default Re: version/source control in datawarehouse project

first,
you can manage your SSIS, SSAS ans SSRS files under a source control system
like visual source safe very easily. so, there is no issue here.

second,
you can use configuration files to store your database connections. each
station need this config file located on the disk and setup for the
developper env. (the config file is an xml file which can be edited by each
developper to reflect the right servers names)
make sure you use a folder like c:\myproject\ssisconfig which is created
exactly the same way on all desktops.
(there is other options to store and share the connections, but this one is
easy to implement)

third,
for the version of the databases...
you have to create a good process to manage SQL script files to keep
everybody updated correctly. for example, organize this into folders:
/Version1
/Patch1.1
/Patch1.2
....
each folder will contains SQL scripts which modify the database and each
developer as to apply the patches on his own environment.
Visual Studio for Database Professional can do this for you.
But I recommend that only 1 person will be in charge of managing the
database and the changes.
http://msdn2.microsoft.com/en-us/tea.../aa718807.aspx

and finally... communicate! :-)


"YesGoGoGo" <YesGoGoGo@gmail.com> wrote in message
news:1187628671.026950.145670@z24g2000prh.googlegr oups.com...
> Hi!
> I'm a freshman in a datawarehouse company.
> All my coworkers are on-site in user side,such as banks,
> and our development machine is also given by user.
> There is no server or desktop in our company.
> After I join this company,I find there's no version/source control in
> our company's projects.
> Because I used to work in a web-based system development company,
> I think it's VERY important to have code under version control.
>
> In my new company,there are 1~3 members in each project.
> Everyone of them will terminal to server,open the same project and
> modify it.
> I try to think if there is a better way to improve this develop cycle.
> First of all,I have to have a version control server,such as
> subversion.
> Because we don't have a server in our company
> ,maybe installing SVN on development server is a good way.
> When sb wants to modify the code(SSIS/SSRS),
> he has to update his working copy to the latest version and modify it.
>
> But for me,there is a problem
> take SSIS for instance, how do we design datasource connectiion to
> make it available for every developer?
> if datasource is DB connection,I can change "localhost" to servername
> or ip to ensure connections on everyone's computer works.
> but for development server,would using servername instead of
> "localhost" make db connection lower?because it has to parse the
> servername? this is my first convern.
>
> If datasource is file connection,there is a similar situation,using
> relative path could make the code fit everyone's working environment
> but maybe inefficient when reading file?
>
> In the begining,I think maybe we can copy DB or files to developer's
> machine, but I find DB and files are too large(too much data...)
>
> And there are many abosolute path in our code
> So if I want to bring version/source control into this project,it
> seems we have to modify those path.
>
> After ETL,we have SSAS,SSRS to do.
> I want to know if there is a better way or best practice to design the
> version control of the datawarehouse project?
> For me,it seems like a mess now...
>
> Need some advice
> Thanks a lot!!
>

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #3 (permalink)  
Old 02-27-2008, 07:08 PM
YesGoGoGo
 
Posts: n/a
Default Re: version/source control in datawarehouse project

thanks to Jeje
your advice is very useful for me!

I've tried to build an environment like what you mention
but there is still a problem about connection configuration.

the following is my SSIS project file
DTS_MIGRATE
AC
BACKUP
DataSource
DOWNLOAD
DP
FD
LN
OTHER
I gather all datasources(*.ds) in DataSource folder and put *.dtsx
into their corresponding folders
So if the db connection on developer's machine is different from
server's
All he has to do is modify *.ds, right?
But I've found in every *.dtsx
db connection is also written in dtsx, like <DTS:Property
DTS:Name="ConnectionString">Data Source=localhost;Initial
Catalog=DW;Provider=SQLNCLI.1;Integrated Security=SSPI;</DTS:Property>
that means developer has to modify all conn string in all dtsx....
that's so frustrating...
Can't I just modify one place(*.ds) and make all dtsx work?

there's another question about edting xml
If I use ultraedit to modify *.ds db connection, although the xml has
changed and saved
After I open the project,these connecitons don't work
I still have to modify these conn in IDE environment

Maybe I'm not familiar to SSIS.

still need help!
thanks a lot!


On 8 21 , 8 16 , "Jeje" <willg...@hotmail.com> wrote:
> first,
> you can manage your SSIS, SSAS ans SSRS files under a source control system
> like visual source safe very easily. so, there is no issue here.
>
> second,
> you can use configuration files to store your database connections. each
> station need this config file located on the disk and setup for the
> developper env. (the config file is an xml file which can be edited by each
> developper to reflect the right servers names)
> make sure you use a folder like c:\myproject\ssisconfig which is created
> exactly the same way on all desktops.
> (there is other options to store and share the connections, but this one is
> easy to implement)
>
> third,
> for the version of the databases...
> you have to create a good process to manage SQL script files to keep
> everybody updated correctly. for example, organize this into folders:
> /Version1
> /Patch1.1
> /Patch1.2
> ...
> each folder will contains SQL scripts which modify the database and each
> developer as to apply the patches on his own environment.
> Visual Studio for Database Professional can do this for you.
> But I recommend that only 1 person will be in charge of managing the
> database and the changes.http://msdn2.microsoft.com/en-us/tea.../aa718807.aspx
>
> and finally... communicate! :-)
>
> "YesGoGoGo" <YesGoG...@gmail.com> wrote in message
>
> news:1187628671.026950.145670@z24g2000prh.googlegr oups.com...
>


Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
  #4 (permalink)  
Old 02-27-2008, 07:08 PM
Jeje
 
Posts: n/a
Default Re: version/source control in datawarehouse project

no, don't use DS files
use configuration files (.dtsxconfig)
right click in a package, then select the configuration option and you'll
found a wizard to create these files for you.



"YesGoGoGo" <YesGoGoGo@gmail.com> wrote in message
news:1187839833.239317.268770@l22g2000prc.googlegr oups.com...
> thanks to Jeje
> your advice is very useful for me!
>
> I've tried to build an environment like what you mention
> but there is still a problem about connection configuration.
>
> the following is my SSIS project file
> DTS_MIGRATE
> AC
> BACKUP
> DataSource
> DOWNLOAD
> DP
> FD
> LN
> OTHER
> I gather all datasources(*.ds) in DataSource folder and put *.dtsx
> into their corresponding folders
> So if the db connection on developer's machine is different from
> server's
> All he has to do is modify *.ds, right?
> But I've found in every *.dtsx
> db connection is also written in dtsx, like <DTS:Property
> DTS:Name="ConnectionString">Data Source=localhost;Initial
> Catalog=DW;Provider=SQLNCLI.1;Integrated Security=SSPI;</DTS:Property>
> that means developer has to modify all conn string in all dtsx....
> that's so frustrating...
> Can't I just modify one place(*.ds) and make all dtsx work?
>
> there's another question about edting xml
> If I use ultraedit to modify *.ds db connection, although the xml has
> changed and saved
> After I open the project,these connecitons don't work
> I still have to modify these conn in IDE environment
>
> Maybe I'm not familiar to SSIS.
>
> still need help!
> thanks a lot!
>
>
> On 8 21 , 8 16 , "Jeje" <willg...@hotmail.com> wrote:
>> first,
>> you can manage your SSIS, SSAS ans SSRS files under a source control
>> system
>> like visual source safe very easily. so, there is no issue here.
>>
>> second,
>> you can use configuration files to store your database connections. each
>> station need this config file located on the disk and setup for the
>> developper env. (the config file is an xml file which can be edited by
>> each
>> developper to reflect the right servers names)
>> make sure you use a folder like c:\myproject\ssisconfig which is created
>> exactly the same way on all desktops.
>> (there is other options to store and share the connections, but this one
>> is
>> easy to implement)
>>
>> third,
>> for the version of the databases...
>> you have to create a good process to manage SQL script files to keep
>> everybody updated correctly. for example, organize this into folders:
>> /Version1
>> /Patch1.1
>> /Patch1.2
>> ...
>> each folder will contains SQL scripts which modify the database and each
>> developer as to apply the patches on his own environment.
>> Visual Studio for Database Professional can do this for you.
>> But I recommend that only 1 person will be in charge of managing the
>> database and the
>> changes.http://msdn2.microsoft.com/en-us/tea.../aa718807.aspx
>>
>> and finally... communicate! :-)
>>
>> "YesGoGoGo" <YesGoG...@gmail.com> wrote in message
>>
>> news:1187628671.026950.145670@z24g2000prh.googlegr oups.com...
>>

>

Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On
Forum Jump


All times are GMT. The time now is 09:27 AM.


Powered by vBulletin® Version 3.6.5
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.2.0
www.UnixAdminTalk.com